|Methods for the Evaluation of the Impact of Food and Nutrition Programmes (United Nations University - UNU, 1984, 287 pages)|
|1. Basic concepts for the design of evaluation during programme implementation|
Both for planning and evaluation, it is important to distinguish between different population groups. The main groups of concern are as follows:
If programme staff have contact with the recipients, obtaining data on these may be relatively easy. This would be the case in the example of a feeding programme, but maybe not in, say, a water supply project. If outcome data are available from recipients these can, to a limited extent, substitute for survey data on the population as a whole.
The distinction between population groups allows construction of a series of 2 x 2 tables that lead to some important indicators for planning and evaluating targeting, as shown in Figure 1.1. (see
The concepts of coverage and focusing have commonsense meanings, both for planning and evaluation. "Coverage." a basic value that needs to be manipulated for different programme designs is equivalent to sensitivity in the epidemiological literature (14); evidently the aim is to optimize coverage. Focusing, which is equivalent to positive predictive value (14) is a less familiar concept. If targeting is to focus resources, focusing should be at least greater than the prevalence in the population as a whole. That is, the proportion of needy in the targeted population should be greater than the proportion of needy in the population as a whole; the same could apply-but is seldom to our knowledge done-for any evaluation of "poverty orientation". There are a number of procedures for choosing appropriate indicators and their screening levels to identify proportion of needy, and for efficiently deciding on cut-off points to define needy (see discussion in ).
For evaluation, the delivery is compared with the targeting and with degree of need, in order to generate further indicators, as shown in figure 1.1. This requires determining whether the recipients were in fact targeted and whether they are needy (e.g. malnourished).
An intermediate stage comparing targeted with recipients (part B of fig. 1.1.) gives indicators of delivery, e.g.: c. the proportion of total targeted who are recipients, which should be 100 per cent if the programme is fully implemented; and of leakage, e.g. as: d. the proportion of total recipients who are targeted, or conversely proportion of total recipients who are not targeted. These should be 100 per cent and 0 per cent respectively if there is no leakage to non-targeted groups.
If there is full implementation and no leakage, then the "actual focusing" and "actual coverage" are the same as those planned (see part C of fig. 1.1.). If there is deviation from the plan, then one way of assessing this is to calculate these "actual" indicators, comparing "needy" with "recipients." Again, actual focusing (recipients needy/all recipients) should be at least greater than the population prevalence of needy. For example, if the prevalence of malnutrition in the region served by the programme is 35 per cent, and the actual focusing is 20 per cent, the evaluator is alerted to a serious problem. Even with knowledge of costs, such indicators could give useful means of evaluating process; with costs as discussed in the next section, they could lead to decisions as to whether the programme is within the range likely to given an adequate or acceptable outcome, even if the expected effects on recipients were achieved. A worked example is given in Mason et al. (15, chap. 4).
If data on needy recipients and targeted populations are available from baseline studies, then some conclusions can also be drawn on outcome during programme implementation based only on outcome data on the recipients. This is so if the assumption can be made that the change in outcome variables is likely to be small compared with that in recipients, and if baseline (pre-programme) data are available. In this case, the need for population surveys for evaluation is reduced. This theory is also given in Mason et al. (15, chap. 4).