|Methods for the Evaluation of the Impact of Food and Nutrition Programmes (UNU, 1984, 287 p.)|
|13. Policy evaluations|
David F. Pyle and Mitchel B. Wallerstein
A policy evaluation methodology
Implication of evaluation methodology
Successful evaluation of the policies that facilitate and direct nutrition intervention programmes is a difficult task under optimal conditions. Even when good data exist (or can be mobilized) regarding the clinical dimensions of nutritional status or the economic circumstances of target populations, it still may prove extremely difficult to establish cause and effect relationships. It is perhaps for this reason, and because meaningful policy evaluation often defies quantification, that the policies that underlie nutrition programming have rarely been considered in other than purely instrumental terms (e.g., tons of food aid delivered, number of children fed, etc.), and that what few policy evaluations exist have tended to be descriptive rather than analytical. The result is that programmes undertaken often bear little or no relationship to the original policy conception and/or are continued with little or no accountability for results.
Nutrition policy evaluations are, almost by definition, the opposite of the neat, empirical assessments of nutritional anthropometry or clinical status that are to be found in the public health literature. Policy evaluations are "soft" in the sense that they involve politics and the individual motivations of policy-makers; and experience has shown that these dimensions are difficult to isolate and analyse empirically. These characteristics have been problematic for the scientific community, which is more comfortable dealing with "controlled" - and, therefore, predictable - situations. Assessment of nutrition policy also does not engender enthusiasm or support within the affected bureaucratic and political circles. From the viewpoint of those who have been involved with initiating or implementing policy, it is one thing to determine nutritional impact with the intention of modifying programmes accordingly, but it is quite another to expose to analysis the entire conception or practicability of the undertaking.
For purposes of evaluation, nutrition policies may be disaggregated functionally into three successive, interlocking stages: policy formation, programme development, and programme implementation. Policy formation is a dynamic process wherein the felt needs of various interest groups are reflected in the political arena as officially-sanctioned government goals and objectives. Programme development, in turn, concerns the manner in which policy is interpreted and acted upon to make the stated goals and objectives operational. The final stage of the policy process is programme implementation, which encompasses all of the activities undertaken to fulfill the specific policy mandate in order to deliver services to the intended target group(s).
Clearly, nutrition policy does not always evolve in such a discrete, hierarchical fashion. However, use of this framework permits us to evaluate: (a) stated objectives - i.e. what policy-makers set out to accomplish; (b) programme content- i.e. how the policy was embodied in operational terms; and (c) policy impacts i.e. what the policy actually accomplished. An understanding of all three dimensions is essential, not only in conducting ex post facto assessments of past and present programmes but also in shaping the design of future initiatives.
Nutrition policy evaluations also involve an examination of the degree of "political will or commitment," an oftused but seldom-defined term. As employed here, political will/commitment is intended to mean the capacity and inclination of decision-makers to follow through on rhetorical statements of support, to maintain programme resources in the face of competing demands, and to defend a programme from its critics. To date, little serious effort has been devoted to understanding, much less evaluating, the importance of political will/commitment in terms of programme impact.
Political will/commitment can be measured at each of the stages of nutrition policy. An initial indicator, at the policy formation stage, is the extent to which rhetorical support for a policy becomes embodied as a physical programme, as measured by staffing quotas, office space, and most importantly, funds. Politicians from many nations have expressed rhetorical concern from time to time for problems of hunger and malnutrition, a position certain to enhance an individual's reputation and career at little or no political cost. Yet when it comes to allocating scarce resources, these same people become conservative and equivocal.
The extent of political will/commitment at the programme development stage is indicated by the specific scope and nature of the programmes established, e.g., limited technical approaches versus ambitious social equityoriented programmes. Technical strategies such as food fortification have the advantage of being relatively easy to adopt and implement and of requiring little or no socio-political change. Social equity-oriented programmes, such as multi-sectoral health and nutrition intervention or land reform schemes, will, on the other hand, demand politically- and economically-difficult reallocation of priorities and resources.
At the stage of programme implementation, political will/commitment is indicated by the extent to which programmes are maintained over time. In many situations, it may be far more difficult to defend a programme (and its budget) from its detractors than to gain the "critical mass" of support necessary for initial implementation. This problem is particularly acute in programmes where positive impact data may be delayed (often by many years) for one reason or another, or where a substantial number of similar programmes must all compete within the same resource pool.
The identification and collection of useful and accurate data on social sector programmes such as nutrition have proved to be a substantial obstacles to meaningful policy evaluation in the developing country context. The traditional measurement of per capita income does not, of course, take account of skewed distribution that distorts the true picture of how the poorest segments of the population actually live. A more accurate view of a government's commitment to, and involvement in, issues of social equity must therefore examine not only the pattern of income distribution, but also other indicators such as educational performance and health status.
The most widely-used indicator of social development is the Physical Quality of Life Index, or PQLI (1) which rates countries according to their level of literacy, infant mortality, rate and life expectancy. It is revealing to compare PQLI ratings with per capita incomes. In the early 1970s, for example, countries such as South Africa and Iran had similar per capita income (about US$1,200 per capita), but ranked along with such developing countries as Burma (per capita income of US$105) and India (per capita income of US$133) in terms of PQLI (53 for South Africa, 43 for Iran, 51 for Burma, and 43 for India). The oil states present an even more unbalanced picture. For example, the richest country, the United Arab Emirates, had a per capita income of US$15,368 but a PQLI of 34, ranking 112 out of 150 countries. On the other hand, a poor country like Sri Lanka (per capita income of US$126) had a PQLI of 82, indicating a high level of social development and equity.
What can a policy evaluation derive from analysing a country's PQLI? Several possibilities arise; for example, in a country with a low PQLI and a high per capita income, one explanation is that there is little political support for, or popular interest in, social development. As a result, the status of the lower socio-economic strata is unlikely to improve no matter what outside assistance is provided. An alternative explanation is that the country is taking measures to improve the social development status of the poor, but that results are not yet measurable in terms of the PQLI.
The evaluation of policies and programmes designed specifically to address nutrition as a social development issue will generally require the mobilization of data from a much broader variety of sources than the PQLI. These may include some or all of the following: (a) examination of public documents and news accounts, (b) analysis of data contained in government reports and/or studies by external agencies, (c) interviews with key decision-makers and administrators both past and present, and (d) limited collection of new data through field observations. The difficulty, as we noted at the outset, is that there are few serious and careful nutrition policy evaluations extant with which results can be compared. It shall be our purpose in the remainder of this paper, therefore, to make a first attempt at setting forth such a methodology and at indicating its limitations and implications.