(introduction...)

Roy I. Miller and David E. Sahn Introduction
The concept of a built-in evaluation system
Reasons for built-in evaluation
Characteristics of a built-in evaluation system
Implementing a built-in evaluation system
Use of the data for overall programme management
References

Introduction

One reason for holding the conference leading to the publication of this document is the general dissatisfaction with the state of the art in the evaluation of food and nutrition programmes. Previous efforts to evaluate such intervention have had several disappointing outcomes. Often, evaluations have been restricted to a review of the process and procedures employed in the delivery of services because of a lack of available data. When impact data have been available, most evaluations have failed to demonstrate nutritional or health impact, or they have produced inconclusive results. Even in the few cases where nutritional and health benefits have been shown, critics have hastened to point out the methodological weaknesses of those evaluations. (A more detailed discussion of the evaluation methodologies used in recent food-aid evaluation programmes and the findings of those evaluation can be found in 11].) Because of those weaknesses (in data collection, measurement, research design and interpretation of results), different approaches to analysis can reveal competing explanations for the observed outcomes or, in many cases. entirely different outcomes altogether.

Traditionally, nutrition interventions have been evaluated using the basic strategies of social science research. Hypotheses are formulated and an experimental or quasi-experimental design is established and applied to test those hypotheses. The element of this strategy that enables the evaluator to attribute observed changes in nutritional or health status to an intervention is the use of controls: if the participant group fares better than the non-participant group, it is assumed that the programme is the cause. When circumstances preclude identifying a randomly-assigned control population to be compared with a treatment group, it is possible to use statistical controls (multivariate techniques such as regression), reflexive controls (comparisons of the treatment group with itself at different points in time), or other analytical techniques to account for or minimize the effects of extraneous factors. (For an example of a mixed strategy using statistical controls to account for differences between a control group and a treatment group, see [2].

For a more general discussion of the array of quasi-experimental techniques, see [3].)

In field settings, the implementation of these strategies for selecting suitable controls has proven to be a difficult and challenging task. The primary source of this difficulty is the lack of constancy of the "real" world and the inability, outside a laboratory, to maintain experimental conditions for a sustained period of time. Specifically, evaluations have faltered because they have:

• failed to account fully for rapidly changing environmental factors, such as socio economics, political considerations, and physical phenomena, which have an equal or greater effect on nutritional status than the programme itself,
• wrongly assumed that interventions are applied consistently over time when, in fact, they undergo dynamic changes in the type and method of service delivery;
• ignored the natural changes that take place in the target group, such as aging, addition and/or attrition of participants, and spontaneous recovery of the malnourished, which, even in the absence of intervention, would result in significant population-based changes in nutritional status, and;
• overlooked the problems inherent in measuring nutritional status; for example, the choice of standard and "cut-off point" for defining nutritional deficiency.

A consequence of these flaws in evaluation is that there remain numerous competing explanations, other than the programme's impact for changes in the target and/or control groups, whatever the research design and/or analytic methodology. In the literature, common sources of competing explanations, often called "threats to validity," have been catalogued and illustrated. (For a general discussion, see [41. For a discussion related specifically to nutrition, see [5].)

Even more disturbing than evaluations without positive or ambiguous results are the large number of nutrition projects and programmes that are never evaluated at all. Evaluation has been viewed as a threat to programme continuity or as an expense hardly justifiable in light of the need to concentrate resources on service delivery. Rarely has evaluation been viewed as a tool to help learn how to achieve greater nutritional impact. The result is that the potential of evaluation as a means of improving project design and implementation has not yet been realized.

In response to the difficulties in carrying out adequate evaluations in the health/nutrition field, a number of chapters in this publication provide valuable guidance. Specifically, the choice and utilization of measures and indicators of nutritional status, as well as the collection and analysis of nutrition-related data, are discussed in considerable depth. Despite the informative nature of these chapters, there remains considerable uncertainty as to whether a "one point in time" evaluation of an ongoing feeding or nutrition programme can yield useful and conclusive results. even if we overcome the difficulties in measuring nutritional progress alluded to above. While it remains unclear that such an approach will ever yield reliable data or definitive indication of impact, even more disconcerting is the small likelihood that evaluation, as traditionally practiced, will improve project performance, justifying the time and resources expended. An alternative approach to evaluation must be considered in order to minimize the methodological problems discussed above, and, concurrently, to broaden the usefulness of evaluation in the planning and implementation phases of a large-scale intervention.

The concept of a built-in evaluation system

An alternative to more traditional methods of evaluation is the inclusion of a system of self-evaluation as an integral part of an intervention. Fundamentally, such an approach involves the integration of evaluation as an ongoing and indispensable element of the planning and implementation of a project. The concept of built-in self-evaluation has been given a great deal of lip service in the past few years and, recently, has been described in the literature in considerable detail. (A proposed system for P.L. 480 Title 11 programmes in India appears in [6].)

A built-in internal evaluation system conceptually merges the tasks of programme monitoring and evalution. Data are generated throughout the chain of programme events, beginning with the cataloguing of inputs, the substantiation of the delivery of services and, finally. the measurement of nutritional impact. This is done on a continuous basis at all project sites. Such information, to be gathered by project staff (as distinguished from outside evaluation teams), is limited to selected key indicators of project operations and impact. The selection of data elements to be gathered and summary statistics to be computed should be based on a review of the minimal information needs of programme managers and a clear understanding of how each element and statistic will be employed in the decision-making process. A premium is placed on coverage at all project sites and on the collation and presentation of such data in simple and understandable formats that can be translated easily into action by managers and their supervisors. In summary, a built-in evaluation system is a self-evaluation system that provides programme planners, field workers, and managers responsible for implementation continuing feedback on their performance.

An important by-product of an internal system that generates data for use in decision-making in the field is the opportunity to aggregate data from individual sites to assess the performance or the programme as a whole. Thus, the system also provides information for use in promoting rational changes in project design on the grand scale. However, for this to work effectively, the information must not flow only in the one usual direction, up through the management hierarchy. It must also flow back down from the central decision-makers, in the form of feedback, to those deployed in the field. The fact that the data were originally collected by field-level people adds strength to recommendations from the "top" that emanate from such data.

From the viewpoint of a funding organization or other agency interested in allocating resources among projects, the primary objections to internal evaluation systems is that the credibility is quite low. James Austin writes:

"This issue frequently takes the form of an outsider vs. insider dichotomy with the latter presumed to be biased because of vested interest in showing performance." (7)

In response, we suggest, first, that if built-in evaluation systems are framed and used as constructive enterprises. as opposed to threatening outside assessments of competency, the problem of objectivity will be mitigated. Furthermore, the concept of built-in evaluation does not preclude external evaluations. Quite the contrary: the potential for success of outside evaluation is enhanced by the wealth of data already available. It is conceivable that such outside evaluation could take on a far broader scope than now possible, not only because of the quality and quantity of data generated by a built-in system. but because of the sensitivity of programme management to the need for, and usefulness of, evaluation activities.

To illustrate how a built-in evaluation system might serve as the basis of an external evaluation, consider the funding agency that wishes to compare the cost-effectiveness of two programmes in order to better allocate resources. Usually, a team of consultants is fielded. On arrival at the site, they learn that there is a dearth of relevant data and that the data are unreliable, and/or incomplete. Furthermore, they must ask interminable questions about the local situation in order to begin to understand the possible explanations for trends observed in the data. With a built-in monitoring/ evaluation system in place, it would be relatively simple to: (a) assign costs to the inputs that are carefully monitored, (b) consider benefits in terms of the achievement of impact, and (c) relate variable costs that accompany changes in their service delivery system to the variable achievement of impact. Clearly, it would be much easier for decision-makers to glean accurate information from a well-functioning internal evaluation system.

A second major objection to built-in evaluation is that the costs are too high to justify the alleged benefit. We believe that this is a somewhat shortsighted view emanating from a poor conceptual understanding of the activity. If the resources for data collection and analysis are thought of as being taken from the programme for an outside activity clearly this argument is valid. But if real gains are derived from evaluation activities in terms of learning how to get more out of the resources applied to the delivery of services, the question becomes whether or not those gains merit the costs. We hypothesize that this will prove to be the case.

Given the discussion above, it is appropriate to consider the merits of built-in evaluation in greater detail. This we do in the next section. However, like so many other ideas, the divergence between theory and practice remains the greatest obstacle to implementation. The remainder of this paper will attempt to elucidate and make practicable the concept of built-in evaluation.

Reasons for built-in evaluation

In the interest of brevity, we will concentrate on delineating just four of the major reasons for incorporating evaluation activities as a regular activity in a nutrition intervention.

 

Programme Operation Improved

The primary reason for incorporating evaluation activities into a nutrition programme is that the knowledge derived will help programme managers improve the quality of their intervention. At the most basic level, the monitoring of the service delivery system will help sharpen the implementation of the intervention. For example, careful monitoring of the stocks and flows of programme inputs may facilitate rationalization of the flow of supplies from warehouses to project sites. This will help avoid losses due to spoilage, contamination, or deterioration that result when commodities and supplies are overstocked at the community level.

At a higher level, evaluation results provide a basis and incentive for programme redesign. It is our belief that the long-term duration of most interventions guarantees that the environment surrounding each intervention must change. Therefore, the "best" programme design must change too. For example, a poor harvest might reduce food availability, raise prices, and decrease food intake by the poor. The resulting deterioration in nutritional status would signal the need for a temporary increase in ration size in a supplementary feeding programme. Furthermore, components of an intervention may meet with success and, as a consequence, no longer be needed. For example, a nutrition education programme, geared at inducing mothers to use oral rehydration techniques at the first sign of diarrhoeal disease may work so well that attention in nutrition education classes should shift to some other activity, for example, boiling water or home gardening. Responding to changing circumstances, as described in these examples, is contingent upon: (a) noting that a change in nutritional status has taken place, and (b) involving functionaries in the determination of why the change occurred and in the formulation of an appropriate response.

Built-in evaluation further facilitates the use of data to improve programme operations for two good reasons: (a) the data will be available in a more timely fashion than data collected in any other way, and (b) the aura of participation surrounding built-in evaluation at all levels will increase the receptivity of site managers, their supervisors, and even the participants themselves, to modifications in programme activities suggested by the findings of the system. It is far less likely that signals generated by a built-in, self-run evaluation system will be discarded or dismissed as incorrect.

 

Data Quality Improved

By building evaluation functions explicitly into an intervention, programme designers provide a genuine incentive for field workers to collect and record accurate data. All too often, programmes are initiated with a set of forms to be filled out in the field and, in some cases, transmitted to some central office. Field workers rarely understand the necessity and purpose of the forms. More often than not, forms, especially those with anthropometric data, simply clutter the health or feeding center and remain unused. Even when forms are shipped to the central office, field workers soon come to realize that no response is forthcoming. Data collection appears to them to be a futile and cumbersome activity, and all motivation for filling out forms accurately is lost. We have encountered field personnel who merely copy last month's form rather than prepare a new one because they perceive that the forms are useless.

However, if a programme is initiated with a set of forms for collecting limited quantities of data and these are used actively for management at the local level, field workers can perceive an immediate purpose in their efforts. When the data are aggregated at higher levels, with feedback given to the field-level functionaries, there is an even stronger motivation for collecting data properly.

A by-product of generating feedback at higher levels of management is the rapid identification of poorly collected and/or falsified data. The review of trends emanating from a single location by a skilled manager is the best protection against spurious or incorrect data. It is immediately apparent, upon review of longitudinal trends, where the data system is breaking down: any place with inordinately large or surprisingly little change has, in all probability, a worker not processing the data correctly.

 

Quantity of Data Increased

If one hundred people spend fifteen minutes each day collecting data, four people would have to spend over six hours each to collect the same amount of data, assuming that all of the data can be collected at the same location. By incorporating data collection as a routine part of an intervention, the collection procedure is spread out over a far larger number of people and places. Thus, the opportunity for generating additional data, more variables, or more cases on the same variables, goes up dramatically.

Also, by sharing the burden of data collection, the cost goes down. Initial costs, the training of so many workers, are high, but recurring costs are minimal. The task becomes another function performed in the field by people already there, often at no additional cost.

The opportunity to generate longitudinal sequences of data on individuals through a built-in evaluation system offers another substantial benefit. The measurement techniques available in the field for ascertaining nutritional status are inherently weak. Also, nutritional status is a dynamic condition that can change rapidly in the face of adversity or improved circumstances. The chance to review the velocity of growth in individuals, in reference to a standard, is of considerable importance and becomes possible through an internal data system.

 

Contextual Information Provided

Having played the role of outside evaluator several times, we have become highly sensitive to our inability to interpret locally-generated data because of our lack of knowledge of the local context. To illustrate: during a recent evaluation of a supplementary feeding programme in Sri Lanka, we encountered an unanticipated result. The nutritional status of preschoolers on the tea plantations, generally assumed to suffer the most severe malnutrition, was better than in the urban, rural, or suburban areas canvassed during our activities. A visit to the tea plantation from which over half the cases in the sample were drawn revealed that this plantation had the model health facility for plantations in the country. (The plantation manager was highly talented and truly concerned with health issues.) Moreover, the medical practitioner in charge noted that the infant mortality rate, even at this model clinic, was abnormally high, because the most malnourished infants were not surviving. Had we not learned of these quirks in our sample, we would have had to challenge the accepted notion that malnutrition was more prevalent on the tea estates. We would have been wrong.

Understanding the local context is critical for proper interpretation of locally-generated data. Even when statistically valid sampling techniques are used to generate programme-wide samples, contextual information knowledge of programme selection procedures, economic trends, impact of parallel programmes, and so forth is needed for proper interpretation. Observation of the local context is heightened when analysis begins at the local level, as it must be with a built-in evaluation system.

Characteristics of a built-in evaluation system

As a starting point for describing a built-in evaluation system. we can identify three components of such a system: the data, the analytic methodology. and the management support structure. Then, having described these basic components, we can offer several principles to guide their design.

 

Components

Data System

The underpinnings of a built-in evaluation system lie in the data collection and recording procedures. Analytic results can be no better (more accurate) than the data used in the computational algorithms. To be effective, a data system should include two types of indicators, impact and process indicators. Measures of impact are needed to determine the degree to which a programme is achieving its goal. Process indicators are needed to ascertain the provision of inputs, their costs, as well as the quality and consistency of the service delivery system. Taken together, it becomes possible to relate project activities to impact. Traditionally, processoriented data have been the subject of project monitoring systems, but have been divorced from any attempt to substantiate or explain the achievement of impact. An example of an impact indicator for a supplementary feeding programme might be the percentage of two-year-olds below 70 per cent of a weightfor-age standard, while a process indicator might be the number of kilograms of the supplement distributed each month. (Note that these indicators are not necessarily directly measurable: they may be computed from simpler data elements stored in the data system. To illustrate, in order to derive the percentage of two-yearolds below 70 per cent of the standard, the age and weight for each child must be ascertained, the weight-forage score calculated, and the percentage of the standard computed. Only then can the overall percentage below some given level thought to define a malnourished state be calculated. Similarly, the amount of food distributed might be computed by subtracting stocks on hand at month's end from the sum of the stocks on hand at the first of the month, plus all shipments received during the month.)

Analytic Methodology

Data are just "numbers" unless a defined procedure for reviewing the numbers is carried out. Although certain analytic procedures may seem obvious, it is our experience that considerable skill is required to arrive at the proper interpretation of the statistics used to summarize a batch of "numbers." For example, it is intuitively appealing to compare the percentage of two-year-olds below 70 per cent of the standard at two points in time to estimate impact. However, the experienced analyst will look at drop-outs in the intervening period (are the malnourished disappearing from the programme rolls faster than the well-nourished?) as well as new registrants during that period (were the new children entering the programme actually better off?). Seasonality, a change in the economic system, or bad harvests might also account for changes in the nutritional impact indicators over time. Similarly, distribution estimates may be abnormally high due to spoilage or pilferage.

An analyst must be trained to review the competing explanations for observed changes in outcome measurements (both impact and process} and to accept only those that withstand an effort at discreditation. It is difficult to conceive of a data system that will systematically collect data on all possible alternative explanations in a "real-life" social setting. Thus, the burden of identifying the most plausible competing explanations falls on the local staff who are living in the area and aware of the changing conditions in their communities. Furthermore, a sense of timing must be introduced; that is, the analyst should learn to wait until trends become clear and not draw conclusions precipitously.

Management Support Structure

To be effective, a built-in evaluation system should be supported at the local level with expertise drawn from higher levels of management. Ordinarily, a hierarchical organizational structure - one calling for the supervision of several local distribution centers by someone of a higher authority and/or greater responsibility (and often a higher educational background or more thorough training - oversees an intervention. The supervisory function provided at these higher levels is very important: first, to provide assistance to the manager of each distribution center in the analysis of data, and second, to transfer knowledge derived at one center to the managers of other centers. Our observation of existing programmes suggests that this mid-level management is often lacking in practice though existing on paper. (In the developing world, the logistics of moving supervisory personnel from center to center often preclude a viable supervisory activity.) Such a situation would totally disrupt a built-in evaluation system.

We suggest that supervisors be well versed in the principle of "management by exception". In reviewing data collected at distribution centers. supervisors should identify centers with abnormally bad indicators or extraordinarily good ones. The former need extra help; the latter may hold the keys to success. (Incidentally, centers with abnormal indicators on either end of the spectrum are often those making egregious errors in data collection and recording). By singling out centers performing on the extremes. supervisors can direct their efforts where they can best be used and, simultaneously, gain insight into what project components or community characteristics lend themselves to the attainment of objectives.

 

Principles

We now offer six principles to guide the design of the data component, the analysis component and, finally the management structure.

1. Data should be generated routinely at the local level. The very name "built-in evaluation system" suggests that the data collection procedure originate at the point of service delivery i.e., in targeted supplementary feeding programmes at the center or clinic performing the targeting function, delivery of health and nutrition services, and/or distributing the food. Although it is possible to conceive of a system relying on data generated by a traveling team of survey specialists, such a team would lack familiarity with the context from which the data were drawn. Routine local data collection, on the other hand, is done by those with greatest knowledge of local conditions and, therefore, the best ability to detect irregularities in the numbers and to interpret the results.

2. The data collected must be used for management of the local center or clinic as well as for evaluation of the programme as a whole. It is not uncommon practice in many existing programmes, especially ones attempting to target services to the most vulnerable, to weigh children as a prerequisite for inclusion in the intervention. In some cases, the data are used as a diagnostic tool to determine what additional services are required by the recipient. But, it is rare, indeed, that data on individuals are aggregated for a service center or health clinic to determine the overall impact of the package of services provided. in the absence of such a management-oriented activity, data systems tend to deteriorate, particularly as humanitarian interests, political pressures, or the press of time force practitioners to abandon data collection activities for the sake of delivering more service.

Aggregation of the data facilitates better management. For example, if it is discovered that two-year-olds are consistently in greater need of, and more responsive to, food supplements than are children of other ages, good managers may consider altering their targeting criteria. They may choose to concentrate their efforts more on two-yearolds or introduce community outreach to find two-year-olds not yet in the programme. It is reasonable to believe that once managers perceive the usefulness of data for sound management, they will continue to take care to collect the data accurately and completely.

3. The quantities of data recorded and analysed for any purpose should be kept to a bare minimum, particularly at the initiation of the system. Because data collection and analysis for any purpose is costly. both in time and money, it is important not to overburden the intervention with data-related chores. Too much data can prove to be as harmful as none at all. When food distribution center managers are inundated with data, they are unable to apply any quality control to the collection and recording activities; they are also less likely to look at and analyse what information is available because of the immensity of the task. A system calling for the examination of a few key indicators is, therefore, a workable and appropriate starting point. The design problem and creative task is selecting the proper summary statistics, given the components of the intervention and the skill levels of the managers. It should be anticipated that the initial system design will prove inadequate quickly as questions of interpretation are raised at the local levels. Therefore, provision for expanding the system should be included in the initial design. However, expansion should be dictated by the experience in the field and not the conventional wisdom of outside consultants.

4. Analytic procedures must be well-defined and understood at the local level. The interpretation of changes in selected impact and/or process indicators can be very difficult. For example, a drop in the supply of a food supplement at a center can be due to many factors: spoilage, an unanticipated increase in the number of programme participants, a failure of a shipment to arrive, and so forth. The manager must be sufficiently skilled to recognize that the change in supply is important, to search for the reason, and to take corrective action if needed. At a more complex level, Community-level functionaries must carry out not only tasks such as weighing children, but also more complex functions such as understanding the rudimentary implications and meaning of such data. That is, the village-level worker must be able to interpret the growth parameters being collected and impart such knowledge to project participants. Furthermore, workers must be trained to perform simple aggregation of data at the village level. This facilitates the recognition and anticipation of communitywide trends. But of even greater importance, when we say "well-defined and understood," it is meant that the manager must be able not only to carry out the computations, but also to identify and examine multiple explanations of the results. This suggests that an area for intense training must be the analysis of data. Training should not stop with form filling-out exercises.

5. The staff implementing the intervention must be committed to goal-oriented management. Too often, programme implementers consider the intuitive argument that feeding hungry people is inherently beneficial, and that is sufficient grounds for carrying out a supplementary feeding programme. Unfortunately, the evidence does not often validate this argument. Therefore, a built-in evaluation system must be based on the notion that a programme, as initially defined, might not reach its stated goals. As a result, managerial talent must be employed at all levels of programme design and implementation to use data to verify and/or facilitate the attainment of goals. This necessitates the abandonment of doctrinaire preconceptions and the substitution for them of an attitude that fosters creativity through recognition that the attainment of goals requires a process of iterative learning and experimentation.

6. The built-in evaluation system must be thought of as a dynamic entity subject to evolutionary growth throughout an intervention. We have already alluded to the need to initiate the system with a minimal set of data and a comprehensible set of analytic procedures. The system will grow as managers at all levels of the organizational hierarchy perceive the need for additional information to interpret the basic indicators. But, more importantly, the system will grow in response to the more sophisticated questions asked by managers once the simpler questions are answered satisfactorily. For example, a system might be designed, first and foremost, to verify that a package of services has had a desired impact. Once that is shown, it is logical to ask which components of the package are "cost-effective." This will require a modification of the built-in evaluation system. In this way, the system will evolve over time in response to the needs of management of the intervention.

Implementing a built-in evaluation system

The previous sections of this report present the arguments, principles and guidelines for built-in evaluation. If the guidelines are followed, we have little doubt that creative programme personnel can formulate internal evaluation systems which will promote iterative experimentation with programme options in the field and, in turn, facilitate the attainment of desired impact. Nevertheless, because built-in evaluation procedures as described above are so rarely applied, we feel it necessary to expand on the specifics of such a system to help others initiate prompt implementation.

Therefore, in this section of this chapter, we will outline one possible starting point for a built-in evaluation system. We emphasize that this is only one of many possible options for internal evaluation systems. Furthermore, we stress that any such system must be contextual. That is, any built-in evaluation system is a function of: (a) the underlying theory or framework which describes how the project intends to achieve impact, (b) the socio-economic environment setting the context of intervention, and (c) the availability of capital and human resources to administer the system. While our example is predicated on the "typical" supplementary feeding programme, it is important to recognize that no actual programme is "typical."

 

The Hypothetical Feeding Programme

A pre-eminent type of feeding programme. and the focus of this workshop, involves the provision of a food ration to vulnerable groups - mothers, infants and young children. The goal of such a programme is to improve the nutritional status of the participants. The role of the food ration is central in this type of programme, for its direct effect on nutrient intake and/or its indirect effect as an incentive for participation in complementary activities (immunization programmes, educational programmes, and so forth).

For any built-in evaluation system to assess the degree to which the goal of improved nutritional status is attained, it is necessary, as an absolute minimum, to measure nutritional status. Furthermore, because of the central role of food supplementation in the intervention, it is necessary, as an absolute minimum, to monitor the stocks and flows of food commidities. In a "straight" feeding programme (one in which food distribution is the only or primary component), these two indicators - nutritional status for impact and commodity stocks and flows for process - may be all that is needed. Ancillary services would require additional impact and process variables. For example, an immunization programme might require the incidence of selected diseases as an impact indicator and the number of vaccines administered as a process indicator. Similarly, major causes of poor nutrition in the environment, which are exogenous to the project services, might suggest other variables to be included in the system to help distinguish programme effects from the secular changes which take place in villages themselves. For example, in an area where shortfalls in agricultural production lead to nutritional deficiency because of resulting food scarcity and employment cutbacks, it may prove useful to monitor agricultural production or food prices.

In the example being developed in this section, it is assumed that the food distribution is directed at preschoolers and is accompanied by an immunization programme. Furthermore, we assume that the distribution is handled through a medical facility where easily diagnosed diseases are identified (e.g. diarrhoea) and treated using simple techniques (e.g., oral rehydration salts). We will ignore environmental factors. However, the analyst who reaches conclusions on the basis of process and impact indicators should acknowledge the role of those factors when presenting evaluation findings.

 

The Data Collection: Forms and Procedures

The primary impact indicator for this system must be a measure of nutritional status. With regard to the selection of the best indicators of nutritional impact, other chapters of this publication discuss in detail many of the available options. We will not reiterate the arguments presented in those chapters. For simplicity, it is assumed that anthropometry is selected for our hypothetical feeding programme. We further limit consideration to weight-for-age. The fact that it is the most common indicator available throughout the world today, coupled with the relative ease of measurement protocol and interpretation of the data, commend the use of weight-for-age in a built-in system.

To determine the ratio of a child's weight to an age-specific reference standard, it is necessary periodically to record the weight of each child and his/her age. The most common device for recording such data is some form of growth chart or "road-tohealth" card which features a graphic representation of a child's growth over time. This device is particularly useful if it is used to demonstrate a child's progress to his/her mother. Figure 14.1. is a sample "road-to-health" chart. This particular chart is based on the NCHS-CDC standard (8) and the Gomez classification; that is, Grades I, II and III malnourishement are taken to be 85 per cent, 70 per cent and 60 per cent of the reference standard (9). "The normal line", the 50th percentile of the NCHSCDC standard, is the imaginary curve between the top two curves on the graph. It is the average of the standard for boys and girls as reported by the CDC.

The inclusion of this chart in our example is not to suggest it as a model to be duplicated elsewhere. While we feel the NCHS-CDC standard is applicable in most, if not all countries, the use of a WHO type chart (10) which does not distinguish among Grades I, II and III may be equally appropriate. In either case, the focus of the nutrition education and growth monitoring effort with the individual mother should be in assuring the child gains weight. Nevertheless, the different grades represented in the chart portrayed are useful in the context of examining age-specific changes in the nutritional status of populations through a series of crosssectional surveys. This, combined with the popular use of a Gomez-type classification, make it an appropriate growth chart on which to base our illustrations.

The chart shown in figure 14.1. (see

FIG. 14.1. Child's Growth Chart) also contains space for recording additional data about the child. Fixed items (name, sex, date of birth, and an identification number) are recorded on the top of the form. Monthly data on the occurrence of disease is recorded just below with room enough for two entries per month. A record of receipt of ration is kept at the bottom of the weight chart. Below that, a record of immunization history for the child is maintained, as is a record of the administration of special treatment. This type of information is often kept on the back of a growth chart. Doing so allows the vertical axis to be lengthened, thereby easing the task of filling out and interpreting growth curves. Nevertheless, the important point is that the "roadtohealth" chart should be explicitly designed to contain all or as many of the data elements for individual preschool children necessary for the determination of (a) impact on a given child's health and (b) impact on the community as a whole, which may be determined through the process of aggregation of individual charts. Before considering how these community-level indicators are derived, let us look at the basic form for recording process data, the stock and flow form.

Month: _________________________
Village: ________________________
Filed By: _______________________

 

Number of children served:

_________ x (ration per child in kgs.)
_________ = Food distributed

STOCK BALANCES

Item	Opening Balance	Receipts	Total	Distributed and Losses	Closing Balance	Call Forward
Food (kgs.)
Oralite (Packets)
Vitamin A (doses)
Deworming (doses)

IMMUNIZATIONS

Vaccine	Number of Children	Number Having Completed Series	Required In Coming Months
BCG
DPT
POLIO

Unlike the weight chart, which is child-specific. the Monthly Inventory Report is maintained for the food distribution center. In short, it is a summary of the stock position for all programme inputs, at both the start and the conclusion of a designated reporting period (e.g., one month). In the sample form shown in figure 14.2. (see FIG. 14.2. Monthly Inventory Report), there is an inherent set of checks and balances. Stock positions at the end of each month are determined by subtracting the quantities of each programme input administered from the starting balance (plus any new stock received during the month). Similarly, at the end of each month, the stock position should be "checked" by counting the stocks on-hand. Differences are indicative of losses due to spoilage, theft and/or poor accounting.

The form shown assumes that the managers of the distribution center order additional quantities of inputs, as needed. We are aware that most supplementary feeding programmes do not allow variable shipments of inputs on a monthly basis. However, in the ideal case, such flexibility would enable management to react to the specific problems in their areas by increasing (or decreasing) the quantities of inputs, on hand, as needed rather than curtailing distribution (or dumping valuable commodities) because stocks are not in balance with requirements. This form can serve as an order form for that purpose. The flows are reported as well as the stock positions so that supervisory personnel, at higher levels can determine if commodities (particularly medicines) are actually being used. For example, if the opening and closing balances indicate identical amounts of oral rehydration salts in stock and no new inventory is added. one can guess that the staff at the center is not diagnosing diarrhoeal disease with sufficient care, or not being sufficiently aggressive in encouraging the use of oral rehydration therapy.

To complete the Monthly Inventory Form, it may be necessary to maintain continuous records on some elements in the form throughout the month. To compute the number of children vaccinated against polio in a month. one could page through all the weight charts and count the children who received polio vaccine during that month. However, it would be far easier to use a "work-sheet" (blank copy of the form) to keep a running tally on polio vaccinations administered. Such a tally can be kept by making hachure (or check) marks in the appropriate box whenever a child is vaccinated. Then, at the end of the month, one need only count the marks.

Forms similar to the "road-to-health" chart and the Monthly Inventory Report are fairly common in the field, but by no means universal. Typically, however, the use of the "road-to-health" chart is limited to diagnosis of the individual child while the record of stocks and flows is used to audit past performance of the delivery system. Neither form is used for programme management. By adding some analysis procedures to those carried out already at the distribution center, these data can be made to "come alive."

Consider first the individual records of nutritional status. By combining the nutritional status of individuals into a summary statistic, the status of all participants (comprising a village or other designated target area) can be estimated. A word of warning is in order. If the generation of a summary statistic describing a community's nutritional status is done improperly, the results can be inaccurate and, in some cases, quite misleading. To avoid this, summaries of nutritional status must be agespecific. That is, four-year-olds should be compared only to other four-year-olds, and so forth. Also, in the typical feeding programme, where entry and exit into the programme is an ongoing process, summaries of nutritional status must account for the changing composition of the target population.

The necessity of accounting for age is described more fully elsewhere (11). In brief, in most situations where malnutrition arises due to chronic food shortage and/or environmental hardship, most children pass through a transition period of maximum risk following weaning. In the absence of intervention most children will "score" worst in any computation of their nutritional status during this high risk period (often between 18 and 24 months of age). They will improve naturally as they continue to grow older, passing out of this vulnerable stage of life. Therefore, in the aggregate, two-year-olds often show higher rates of malnutrition that four-yearolds. This natural improvement must be accounted for in any analysis of the change in nutritional status of a community over time.

The entry and exit of participants can cause marked changes in the nutritional status of the community as a whole, if either the drop-outs or the new enrollees, as a group, differ in their nutritional profile from the rest of the community. The extreme example of this is the death of third-degree malnourished programme participants. The number of malnourished children receiving treatment decreases accordingly. Because new enrollees are typically younger than programme "graduates," their nutritional profile is different, due to the aging phenomenon described above. Thus, it is essential to account for these differences in any analysis of the change in nutritional status of a community over time.

There are a variety of statistical techniques which can be applied to account for entry, exit and aging in an analysis of community nutritional status. For its simplicity and practicability in a field setting, we recommend an analysis based on a graphic representation of change over time rather than on a statistical analysis. To illustrate this concept of using the graphic approach in an operational context, we will now describe tables and plots that we refer to as "characteristic curves." One can learn quite a bit about the characteristics of a community from the shape of the curves (11).

Figure 14.3. shows a sample form that may be used in computing the numbers needed to draw the "characteristic curves" depicted in figure 14.4. It is stressed that for each programme the actual forms and field protocol will be different and shall be developed accordingly. In this example the summary data in figure 14.3. are collected every three months. Individuals may be weighed more frequently than prescribed by the aggregation procedure; for example, children may be weighed every month even though the data are aggregated on the Nutritional Status Summary form every three months. Decisions and protocol in this regard are contextual. Just as forms must be tailored to the individual programme, so too should measuring and recording procedures. It is cautioned that any data must be interpreted within the context of changing seasons. Quarterly curves become far more useful in the second and third years of a programme because seasonal trends will become known.

FIG. 14.3. Nutritional Status Summary

VILLAGE:
MONTH:
FILED BY:
CHILDREN WEIGHED IN BOTH THIS AND LAST QUARTER

e	Status of continuing participants
	Normal	Grade I	Grade II	Grade lIl	Total
0-12
13-24
25-36
37-48
49-60
TOTAL
New entrants
0-12
13-24
25-36
37-48
49-60
TOTAL
Droup - outs
0-12
13-24
25-36
37-48
49-60
TOTAL

 

FIG. 14.4. Sample Characteristic Curves and Data on Malnourished Children (the image)

...and the tables

Quarter ending March 31

	0-12	13-24	25-36	37-48	49-60	Tot.
Mall	8	25	35	30	17	115
Tot.	40	96	104	100	60	400
Pct.	20%	26%	34%	30%	28%	29%

Quarter ending June 30

	0-12	13-24	25-36	37-48	49-60	Tot.
Mal.	6	24	31	27	16	104
Tot.	26	100	102	102	64	394
Pct.	23%	24%	30%	26%	25%	26%

Malnourished = less than 70 per cent of the NCHS-CDC standard.

The data in figure 14.3. consists of the count of children by age group, in each nutritional category. The village level worker may use a blank form filled out by making hachure marks, as for the Monthly Inventory Report, continuously during each weighing session to facilitate preparation of the top two sections of the Nutritional Status Summary form.

In the top section of the form the nutritional status of continuing participants is recorded. For new entrants, a new individual weight chart is prepared initially, since these children were not participating in the previous weighings. The data from these charts are aggregated to fill in the middle section of the Nutritional Status Summary. Drop-outs are those not weighed in this quarter who were weighed in the previous quarter. They consist of graduates from the programme due to age, as well as deaths, out-migrants, and so forth.

Drop-outs cannot be counted using check marks but must be found by paging through all of the weight charts. This can be done by going through some sort of looseleaf notebook which contains all the charts, if they are kept at the center. Alternatively, if charts are kept at home with the mother, a roster of all participants can be maintained with space reserved for a notation that a child participated in any given weighing period. Children not checked in any given period are "drop-outs." Such a roster can be an invaluable aid if a census of all potential participants in a geographical area is made in order to screen out those children not needing help. Also, if no-shows are noted on that roster and reasons for not showing determined through community outreach, it becomes possible to estimate mortality and migration rates.

Depending on the size of the distribution center, it may be necessary to use larger categories than those on the sample form. This can be done, for example, by combining Grade II and Grade III or using age categories spanning nine months or eighteen months. Similarly, it is possible to use smaller categories, e.g., six month age groups. This would be preferable.

In a similar vein, if a Grade I, II, or III classification of malnourished is not used, as would be the case with the WHO growth chart, figure 14.3. may have notations such as "gained weight," "stayed the same," and "lost weight" in lieu of the notations Grade I, II and III. The key to the definition of the categories is their stability. One wants to avoid the situation where a change in the status of only one or two children can markedly alter the percentage of children in any age group declared to be malnourished.

With the data in this table, it is possible actually to draw the "characteristic curves" referred to above. These are graphs with the percentage of malnourished children plotted on the vertical axis and age plotted along the horizontal. Figure 14.4. is a hypothetical example of a set of "characteristic curves" for a group of 200 children who participated in a feeding programme and were weighed in successive quarters. These two curves show an improvement in the nutritional status of children in all age categories, except the youngest. The increase in malnutrition among the youngest should not be surprising because these children are fast approaching the age of maximum risk. Note, the total number of children has decreased by six- the number "graduating" during the three month period. We assumed no new enrolments.

Once again, numerous alternatives to the type of characteristic curve in figure 14.4. should be considered. For example, another possibility would be to use "failure to gain weight," and the corresponding percentage, in place of the concept of malnourished (which, in our example, is defined as less than 70 per cent of the NCHSCDC standard).

The successful application of the concept of the "characteristic curve" requires the selection of an appropriate definition for malnutrition and the selection of the best set of curves to be plotted on a single set of axes. We cannot offer a unique set of guidelines for making these selections. Clearly, various contextual factors must guide the decision-making process for any given programme.

Finally, there are a variety of possibilities for plotting different sets of curves, only one of which is illustrated in figure 14.4. Any, or all, might be suitable for a given analysis. For example, one might wish to plot not only the percentage malnourished at a single point in time, but also the percentage in each grade of malnourishment. In circumstances where there is considerable entry and exit in a programme, one might choose to plot the curves for new enrollees and/or graduates to facilitate comparison of their nutritional profile with that of continuing participants. Over a year's time, one might also plot the curves for all four quarters to detect seasonal trends.

A particularly useful pair of curves are the two curves set a year apart. If twelve month age categories are used, the children in one category during the earlier year are the same cohort (the exact same children) as the children in the next oldest category in the latter year. In this instance, once can see both the progress of a particular group of children and the progress of an age grouping in a single graph. (An additional tool for observing the progress of a fixed group of children is the transition matrix- an array of numbers where each element is a count of the children starting out in one grade of malnutrition [given by the row of the matrix] and "moving" to another grade [marked by the column of the matrix!.)

The form in figure 14.3. contains the raw data for any number of "characteristic curves." One need only compute the relevant percentages and plot the curves. Learning to interpret the curves can be difficult, at first, but interpretation becomes relatively straightforward with experience. One must remember, however, to consider changes in the local economy, the local infrastructure, and such things as the initiation of other social programmes before attributing changes in the nutritional status of the community to the intervention alone.

Use of the data for overall programme management

Thus far, we have considered a village-oriented data system consisting of a record of the stocks and flows of programme inputs and a computation of community-level impacts from data gathered on individual participants. Managers of village-level operations can make use of this data in making decisions concerning their own day-today operations. If the data is passed up to higher levels of management within a large-scale programme, the data can help guide decision-making at those supervisory levels as well. If the summary forms (figures 14.2., 14.3. and 14.4.) are transmitted from the field to management centers, it will be unnecessary to transmit the "road-tohealth" charts on individuals.

One useful way to accomplish higher-level management in the system is through aggregation or consolidation of the data for larger areas or regions. For example, suppose a regional supervisor is responsible for programmes in twenty villages. By using "characteristic curves" made up from the totals derived by adding the numbers on the Nutritional Status Summary forms for all twenty villages, the supervisor can delineate trends in his/her own territory. Of course, the identification of such trends is most useful at a level of management where programming decisions can be made. It does no good for a supervisor to discover that the critical age for preventive feeding is 20 months, if he is forbidden by fiat from above to feed children less than 2 years of age.

A second and, perhaps, more useful way to use data describing the performance of individual distribution centers is to compare the events and trends among those centers to find the exceptional cases. This process was described earlier as the application of the principle of management by exception. If a regional manager recognizes that one village is not using oral rehydration salts (as reported on the stock-flow form) and is showing the smallest nutritional improvement (as shown on the "characteristic curves"), he/she can take steps to correct the procedures in that village or, at least, inquire into the reasons for the lapse in treating diarrhoea at that location. Similarly, consider the situation where progress at one distribution center is extraordinary. If upon further inquiry it is found that the local staff on their own have instituted an educational programme, on the importance of eliminating sharing of the ration within the family, this suggests the need for similar innovative education programmes elsewhere.

There is always some risk involved in providing quantitative information to decision-makers. Poor managers tend to rely solely on the "numbers" rather than on first-hand observation of the processes they control. We are not suggesting that the quantitative information replace sound judgment based on a host of supporting information. In the ideal situation, the analytic results will act as signals which trigger additional in-depth inquiry by good management. Then, the data and the results of such inquiry can be used together to promote better programme operations and design.

Finally, the data base created by the type of system outlined above is ideal for both internal and external evaluation work. Of particular interest is the potential of such a system for promoting analytic responses to evaluation research and/or programme design questions. The possibility for field testing propositions concerning intervention which, heretofore, have been debated in the abstract by the "experts" is unbounded.

An example of such an issue is the ration size for a given programme. By altering the ration size in selected villages, the built-in evaluation system will generate data, automatically, on the impact of the new ration size. Historical trends will exist for the selected villages facilitating longitudinal analyses. Simultaneously, trends in other, similar villages can be mapped in parallel with those in the selected sample. Thus it is possible to set up classical experimental designs without having to initiate special data gathering activities.

References

1. David Sahn, Principles and Guidelines for the Evaluation of P. L. 480 Title II (Food-for-Peace Programmes and Policies), (Community Systems Foundation, Ann Arbor, Michigan, 1981).
2. Roy I, Miller and William D. Drake, "A Revisitation Two Years after Programme End." part 11 of The Promotora Programme in Candelaria (Community Systems Foundation, Ann Arbor, Michigan, 1980),
3. T.D. Cook and D.T, Campbell, Ouasi-Esperimentation (Houghton Mifflin, Boston, 1979).
4. Donald T. Campbell and J.C. Stanley. Experimental and Quasi-Experimental Designs for Research (Rand McNally, Chicago, 1966), pp, 5-6.
5. David E. Sahn and Robert M. Pestronk, "A Review of Issues is Nutrition Programme Evaluation", A,l.D, Programme Evaluation Discussion Paper No. 10, (USAID, Offices of Nutrition and Evaluation, Washington D.C,, 1981), pp, 42-51,
6. Roy I. Miller and David F. Pyle, Toward a Monitoring and Evaluation System for P.L. 480 Title II Maternal Child Health Programmes in India (Community Systems Foundation, Ann Arbor, Michigan, 1981).
7. James E. Austin, "The Perilous Journey of Nutrition Evaluation," American Journal of Clinical Nutrition, 31: 2327 (1978),
8. Peter V.V. Hamill, Terence A, Drizd, Clifford L. Johnson, Robert B, Reed and Alex F, Roche, NCHS Growth Charts, 1976, Monthly vital statistics report, vol, 25 No. 3 supplement (National Center for Health Statistics. US Department of Health, Education and Welfare, Rockville, Maryland, 1976).
9. F, Gomez, C,R. Cravioto and J. Frenk, "Malnutrition in Infancy and Childhood, with Special Reference to Kwashiorkor," Advances in Pediatrics, 7 (1955).
10. WHO, A Growth Chart for International Use and Child Health Care: Guidelines for Primary Health Care Personnel (World Health Organization, Geneva. 1978),
11. William D, Drake, Roy I. Miller and Margaret Humphrey, Final Report Analysis of Community-Level Nutrition Programmes (Community Systems Foundation, Ann Arbor, Michigan, 19801, pp. 94-97