|Methods for the Evaluation of the Impact of Food and Nutrition Programmes (UNU, 1984, 287 pages)|
|3. Measuring the impact of nutrition interventions on physical growth|
Mason and Habicht (see chapter 2) warn investigators not to embark on impact evaluations without first examining whether it is reasonable to expect any effects. For example, a feeding programme is unlikely to affect physical growth if only very insignificant amounts of food are actually distributed to the target population. Also, food-for-work programmes of very short duration (e.g., one month) are unlikely to improve the nutritional status of workers and their families. On the other hand, a long-term and major increase in protein and energy intakes in preschool children previously found to have poor diets and marked growth retardation is likely to result in significant changes in health and nutritional status.
Strategies for showing whether nutrition programmes have had an impact on physical growth will vary greatly, depending upon the programme's design and data characteristics. As is repeatedly pointed out by others in this volume, conditions are rarely ideal, as many programmes lack adequate built-in evaluation modules. The first step in assessing whether an impact on growth has occurred is to show that the programme is associated with improvements in physical growth. In the rare situations when baseline data are available, pre-test/post-test differences in the target population will provide a measure of the changes associated with the intervention. As there is often a wide range in the degree of programme participation (e.g.. attendance, time in the programme, amounts received, etc.), researchers should also examine whether these gradients are associated with physical growth.
A second step in the analyses is to show that the changes observed were due to the programme and not to other factors. Data collected for the same time interval on comparison groups (i.e., similar in all important characteristics but not participating in the programme) would be useful for this purpose but are rarely available. Instead, researchers often rely on multivariate analysis to test whether potentially confounding variables account for the observed associations between nutrition variables and physical growth.
In some nutrition interventions, particularly those involving supervised feeding, anthropometric data on those individuals attending the food distribution centres are often collected. These data can be very useful in evaluating whether the programme has improved nutritional status. For example, children just entering the programme can be compared to children the same age who have been in the programme for a longer time. Data showing that the latter are heavier and taller would suggest that the programme is effective. Moreover, if longitudinal data are available, growth rates of participating children can be related to programme characteristics (e.g., attendance).
While useful, as the above examples show, data on just the participants have important limitations. Coverage is often poor and serial measurements on individuals may be few and far between. Those individuals not participating in the programme will obviously not be represented in the sample and, because of variability in participation, there will be a tendency for more frequent attenders to be measured more often. The absence of representative data on the target population will make it very difficult for programme evaluators to ascertain whether the intervention reached those in greatest need and would also make interpretation of the results difficult. For example, an apparent improvement in physical growth when compared to surveys prior to the beginning of the programme might only reflect that attendance was greater for the better educated, wealthier, and better fed families.
Where possible, it is highly recommended that anthropometric data on participants be complemented by cross-sectional surveys in the target population (including participants and non-participants) and in similar populations not participating in the programme (control or comparison groups!.
Ideally, cross-sectional surveys, including at least one during the time before the initiation of the programme (baseline), should be carried out on a regular basis. Because the logistic problems involed in measuring the same individual through time are formidable, the longitudinal approach is not recommended. (In the clinic or feeding centre, where the aim might be to monitor the health and nutritional status of individual children, serial data would, of course, be desired.) In the simple design
TABLE 3.1. A Simple Design for Evaluating the Mean Effects of Nutritional Interventions on Physical Growth at Two Points in Time
Where A1 and A2 are mean values for samples collected at the baseline and intervention phases, respectively, in the comparison population. Similarly, B1 and B2 values are mean values for baseline and intervention phases, respectively, in the test population. No impact would be concluded when A2 - A1 = B2 - B1 Positive impact would be concluded to the degree that B2 - B1 > A2 - A1 Negative impact would be concluded to the degree that A2 - A1 > B2 - B1 shown in table 3.1., target and comparison populations would be measured prior to the intervention (baseline) as well as during it. Ideally, data on several points throughout the intervention phase are desirable in order to characterize the dose-response nature of the relationships. As explained in the footnote to the table, impact would be a function of changes taking place in the target sample relative to the comparison sample.
The use of a control population facilitates the interpretation of the results but raises ethical considerations. Ethical issues become most salient when food or nutrients are available for distribution but are withheld for research reasons. On the other hand, where resources do not allow for full coverage of the needy population, the selection of a control population from the untreated areas would seem to be justified. Ethical issues also need to be examined when collecting baseline data. While a rapid, cross-sectional survey before the program's initiation might be viewed as an ethically acceptable strategy, the seasonality aspects mentioned below would argue for a longer-term period of observation. There are no general answers to ethical questions, and each particular programme needs to be examined carefully. (There is always the possibility that well-meaning programmes will actually produce detrimental results. Hence, researchers would seem to be morally compelled to have adequate research designs and this would require the use of baseline information and of comparison groups.)
Another important issue in cross-sectional studies is seasonality. Because of swollen rivers, deteriorated roads, and other similar obstacles, survey research is more difficult to carry out during the wet season. These difficulties are undoubtedly the reason why data collection in tropical countries is greater during the dry season (1). This is unfortunate because there is a strong seasonal dimension to nutritional deficiencies and infectious diseases in developing countries, the wet season typically being the time when prevalence and severity of many problems are greatest. It is strongly recommended that seasonality be taken into account in drawing up the data collection plans. A satisfactory approach would be to extend data collection over a full-year cycle, or always to collect data at the same time (e.g., after the harvest).
Another issue facing evaluators is whom to measure: all members of the family or only the so-called vulnerable groups: pregnant and lactating women and small children. If limitations of personnel and time are overwhelming, data collection should, of course, be limited to women and children. On the other hand, the collection of anthropometric data on other members of the household may, in conjunction with dietary and energy expenditure data, shed light on how food was distributed within the family. By focusing only on the so-called vulnerable groups, researchers might miss the full picture of what happens in nutrition interventions.
Researchers will always find less-than-ideal situations when evaluating programmes. Nonetheless, it is often possible to derive useful conclusions about the benefits of the programme if appropriate analyses are carried out. At the same time, researchers should be prepared to admit that sometimes a minimally satisfactory assessment of impact simply cannot be carried out. To proceed with the evaluation in such instances could lead to serious errors. Researchers might conclude, for example, that no impact occurred, not because none took place, but because the study lacked the power to detect it.