|Methods for the Evaluation of the Impact of Food and Nutrition Programmes (UNU, 1984, 287 pages)|
|14. Built-in evaluation systems for supplementary feeding programmes why and how|
In the interest of brevity, we will concentrate on delineating just four of the major reasons for incorporating evaluation activities as a regular activity in a nutrition intervention.
Programme Operation Improved
The primary reason for incorporating evaluation activities into a nutrition programme is that the knowledge derived will help programme managers improve the quality of their intervention. At the most basic level, the monitoring of the service delivery system will help sharpen the implementation of the intervention. For example, careful monitoring of the stocks and flows of programme inputs may facilitate rationalization of the flow of supplies from warehouses to project sites. This will help avoid losses due to spoilage, contamination, or deterioration that result when commodities and supplies are overstocked at the community level.
At a higher level, evaluation results provide a basis and incentive for programme redesign. It is our belief that the long-term duration of most interventions guarantees that the environment surrounding each intervention must change. Therefore, the "best" programme design must change too. For example, a poor harvest might reduce food availability, raise prices, and decrease food intake by the poor. The resulting deterioration in nutritional status would signal the need for a temporary increase in ration size in a supplementary feeding programme. Furthermore, components of an intervention may meet with success and, as a consequence, no longer be needed. For example, a nutrition education programme, geared at inducing mothers to use oral rehydration techniques at the first sign of diarrhoeal disease may work so well that attention in nutrition education classes should shift to some other activity, for example, boiling water or home gardening. Responding to changing circumstances, as described in these examples, is contingent upon: (a) noting that a change in nutritional status has taken place, and (b) involving functionaries in the determination of why the change occurred and in the formulation of an appropriate response.
Built-in evaluation further facilitates the use of data to improve programme operations for two good reasons: (a) the data will be available in a more timely fashion than data collected in any other way, and (b) the aura of participation surrounding built-in evaluation at all levels will increase the receptivity of site managers, their supervisors, and even the participants themselves, to modifications in programme activities suggested by the findings of the system. It is far less likely that signals generated by a built-in, self-run evaluation system will be discarded or dismissed as incorrect.
Data Quality Improved
By building evaluation functions explicitly into an intervention, programme designers provide a genuine incentive for field workers to collect and record accurate data. All too often, programmes are initiated with a set of forms to be filled out in the field and, in some cases, transmitted to some central office. Field workers rarely understand the necessity and purpose of the forms. More often than not, forms, especially those with anthropometric data, simply clutter the health or feeding center and remain unused. Even when forms are shipped to the central office, field workers soon come to realize that no response is forthcoming. Data collection appears to them to be a futile and cumbersome activity, and all motivation for filling out forms accurately is lost. We have encountered field personnel who merely copy last month's form rather than prepare a new one because they perceive that the forms are useless.
However, if a programme is initiated with a set of forms for collecting limited quantities of data and these are used actively for management at the local level, field workers can perceive an immediate purpose in their efforts. When the data are aggregated at higher levels, with feedback given to the field-level functionaries, there is an even stronger motivation for collecting data properly.
A by-product of generating feedback at higher levels of management is the rapid identification of poorly collected and/or falsified data. The review of trends emanating from a single location by a skilled manager is the best protection against spurious or incorrect data. It is immediately apparent, upon review of longitudinal trends, where the data system is breaking down: any place with inordinately large or surprisingly little change has, in all probability, a worker not processing the data correctly.
Quantity of Data Increased
If one hundred people spend fifteen minutes each day collecting data, four people would have to spend over six hours each to collect the same amount of data, assuming that all of the data can be collected at the same location. By incorporating data collection as a routine part of an intervention, the collection procedure is spread out over a far larger number of people and places. Thus, the opportunity for generating additional data, more variables, or more cases on the same variables, goes up dramatically.
Also, by sharing the burden of data collection, the cost goes down. Initial costs, the training of so many workers, are high, but recurring costs are minimal. The task becomes another function performed in the field by people already there, often at no additional cost.
The opportunity to generate longitudinal sequences of data on individuals through a built-in evaluation system offers another substantial benefit. The measurement techniques available in the field for ascertaining nutritional status are inherently weak. Also, nutritional status is a dynamic condition that can change rapidly in the face of adversity or improved circumstances. The chance to review the velocity of growth in individuals, in reference to a standard, is of considerable importance and becomes possible through an internal data system.
Contextual Information Provided
Having played the role of outside evaluator several times, we have become highly sensitive to our inability to interpret locally-generated data because of our lack of knowledge of the local context. To illustrate: during a recent evaluation of a supplementary feeding programme in Sri Lanka, we encountered an unanticipated result. The nutritional status of preschoolers on the tea plantations, generally assumed to suffer the most severe malnutrition, was better than in the urban, rural, or suburban areas canvassed during our activities. A visit to the tea plantation from which over half the cases in the sample were drawn revealed that this plantation had the model health facility for plantations in the country. (The plantation manager was highly talented and truly concerned with health issues.) Moreover, the medical practitioner in charge noted that the infant mortality rate, even at this model clinic, was abnormally high, because the most malnourished infants were not surviving. Had we not learned of these quirks in our sample, we would have had to challenge the accepted notion that malnutrition was more prevalent on the tea estates. We would have been wrong.
Understanding the local context is critical for proper interpretation of locally-generated data. Even when statistically valid sampling techniques are used to generate programme-wide samples, contextual information knowledge of programme selection procedures, economic trends, impact of parallel programmes, and so forth is needed for proper interpretation. Observation of the local context is heightened when analysis begins at the local level, as it must be with a built-in evaluation system.