|Improving agricultural extension. A reference manual. (1997)|
|Part II - Improving extension programmes and processes|
|Chapter 11 - Evaluating extension programmes|
Which methods are right for the task? The basic rule here is that the selection of methods follows the selection of focus, not the other way around. Each evaluation question must be examined in relationship to what would constitute evidence for answering it. The following brief descriptions of data collection methods, although by no means exhaustive, can be used as a toolkit for a variety of circumstances. The list includes document analysis, observations, interviews, surveys, focus group committees, field visits and tours, village drama and role plays, maps, case studies, field trial documentation by farmers, remote sensing, and aerial photography.
Document Analysis. Examples include minutes of meetings, correspondence, budget records, workshop notes, participant papers, and newspaper reports, to name a few. These can be treated as data, analysed for content, and summarized in relation to questions, including extent of inputs into the programme; levels of participation, nature of goals and activities, and themes regarding problems, concerns, expectations, and new directions. Themes from documents can be a credible source of information. These documents usually do not reveal participants' motivations or subjective experiences. However, documents often reveal difficulties of programme operation.
Observations. Observers can be outsiders or persons who are involved in learning activities. Observers are usually given a short list of items that may include extent of participation and personal interaction, nonverbal indicators of interest or inattention, leadership roles, performance levels, and conflict indicators, to name a few. Both qualitative and quantitative data can be collected.
Findings can be reported to the learners as a whole to start a reflective process about what may need to be changed, or they can be used as evidence of successful methods or learning outcomes (Worden & Neumaier, 1987). Observations of process and outcomes can be recorded by video or photo documentation. These data are very powerful graphic ways of communicating the nature of a programme and its outcomes to sponsors as well as to potential learners. Video records of learners' local knowledge also can be used to help them reflect on strengths and limitations of knowledge.
Interviews (Key Informants, Oral Histories, Storytelling). Interviews are probably the most widely used method for programme evaluation, including the evaluation of learning. Interviews with key informants and representative farmers are suited to indepth explorations of issues. If the questions are standardized, responses can be tabulated numerically to indicate item strength. If questions are open-ended, indepth unique responses can be generated, which in turn can provide information regarding reasons why the activities are viewed differently by diverse groups of participants. Individual oral histories can inform patterns of practice and the use of extension resources. Storytelling is one of the oldest forms of human social life. Evaluative judgements are usually embedded in stories. Interviews, oral histories, and Storytelling with farmers who are not served is essential to a full picture of extension. Illiterate members of the community can participate fully through interviews, oral histories, and Storytelling, demonstrating that illiteracy does not rule out participatory evaluation.
Group interviews (sometimes called focus groups) can be formed according to geographic locations or by farm type to discuss specific evaluation questions. Sometimes the groups may already exist. At other times new groups may be formed just for the evaluation. The purpose is not only to generate judgements using agreed upon criteria, but also to uncover unanticipated outcomes, applications, opportunities, and problems to inform future extension efforts. A community can reconstruct its history, chronology of events, crises, turning points, accomplishments, and so forth. Chalkboards, pictures, and drawings can represent milestones or decisions. The art of conducting group interviews can be learned. Many community members have a talent for this and are more likely than agency staff to evoke authentic responses. Subgroups are often necessary to listen to voices that are unlikely to be heard in groups dominated by those with status.
Surveys. The survey is a more standardized form of data collection that incorporates a prepared questionnaire. In most parts of the world, surveys of farmers in rural areas must be conducted by interview using interviewers who know the territory, language, and culture of potential respondents. Sometimes surveys can be administered at meetings or public gatherings; however, responses have to be treated as an "opportunity sample," rather than as a "random sample," and therefore generalization of the findings to a larger population is limited. Surveys are often used to evaluate the extent of practice, estimations of production yields, preferences for appropriate technology, and expectations regarding the future. They are best used with homogeneous populations rather than with quite diverse populations because standardized questionnaires are less likely to be sensitive to diversity. Evaluation of unique practices and adaptations of diverse farmer types is best done through interviews and observations.
Field Visits and Tours. There is no real substitute for field visits and tours to provide authenticity and reality to the conditions, limitations, and impacts of extension programmes. Evaluation teams composed of local farmers, extension staff, administrators, funders, and external evaluators provide balance and interactive learning regarding different perspectives. Team members can undertake observations and interviews and learn from each other about their findings during travel. Assignments can be distributed so that special knowledge regarding specific aspects of situations can be gathered. Some team members can focus on economic, social, and cultural aspects, while others focus on technical aspects. Comparing data, analysis and reflection on findings, and insights following field visits can generate a more holistic, balanced evaluation.
Documentation of Farmer Demonstrations. Farm visits, which have been used frequently by extension staff for teaching and the transfer of technology, also can be used for evaluation of appropriate technology. Farmers and local leaders can be taught to conduct their own field trials, thus fostering the pride and dignity of local people who can transfer appropriate technologies through their local languages. When farmers choose their own focus of inquiry, collect and analyse their own data, and come to their own evaluative judgement, they are more likely to adopt and pass on relevant and effective appropriate technologies. World Neighbors NGOs in Bolivia and Peru have developed a standardized means by which farmers engage in site-specific experimental documentation and reporting of crop yield comparisons, using simple calculators (Ruddell, 1994).
Village Drama and Role Plays. Asking farmers in farmer association meetings or in village gatherings to create a drama or role play that describes the interaction process of extension staff with the village on a specific practice will reveal a variety of evaluative data on social relationships, relevance of extension knowledge to local knowledge, and historical events that have affected the solutions to farmer problems.
Maps. The generation of maps can provide a basis for making judgements about access to extension resources by showing where farmer contacts have been made. Maps also can be made to show the location of sustainable agricultural practices. These facts can be used to evaluate the scope and effectiveness of extension efforts on practices in a region, watershed, or geographic area. When access maps are overlaid with social ranking maps, judgements can be made regarding benefits by social class. Concept maps created collaboratively among farmers and extension staff can provide explanations regarding success and failure of specific programme efforts. Reflections on these maps, which can be drawn on chalkboards or in the sand, can reveal contradictions in underlying assumptions and expectations, which in turn can lead to new experiments.
Maps showing before-and-after photographic and remote sensing images can be the basis for evaluative discussion regarding sustainability of farmer and extension staff practices. This is especially relevant to desertification, deforestation, soil erosion, and status of wetlands and wildlife habitat. These data can help direct extension activity to environments that are most at risk, as well as provide evidence regarding extension's positive impacts from collaboration with farmers on sustainable agricultural and natural resource practices.
Case Studies. In order to understand motivation of farmers or potential extension contacts, case studies of specific types of farmers or farming practices can be undertaken. Comparisons between farmers who have used extension technologies and those who have not are common types of case studies. The typology may be based on geographic regions, soil types, and cultural, age, gender, and economic differences. Case studies are best constructed through repeated interviews over time and often include, in addition to self-report, data from persons who know the subjects well. Oral histories, logs, and journals can also contribute to case study data if farmers collaborate in producing these case studies. Evaluators should guarantee the right to privacy and confidentiality of their sources. Case studies often reveal deterrents to participation, as well as ways participants have overcome deterrents to practice through extension contacts.