A policy evaluation methodology

Two basic assumptions underlie the methodology described here. First, it is assumed that the policy evaluations are to be conducted either retrospectively or concurrent with operational programmes, i.e., that the focus is on the origins and ramifications of past and present policies. A second, related, assumption is that the policy evaluations are not intended to be prospective, i.e., that the principal objective is not necessarily the development of new programmatic approaches (although this is certainly one possible outcome) Because the conditions under which policies evolve and are implemented vary substantially, depending on the country context, it is neither possible nor advisable to spell out a "cookbook" approach that can be applied without modification under all circumstances. Rather, we have outlined here a generic methodology that can (and should) be modified and adapted to the specific type of nutrition policy being evaluated.

Our approach has been to set out a series of scales for each of the three policy stages by means of which performance factors can be measured and the impact of a given policy evaluated. When employed successfully, the overall effectiveness of a country's nutrition policy may be determined by comparing actual performance with the potential score. Of course, there is also a problem with the use of scales in attempting to weight the importance of one parameter relative to another; that is, how seriously should a low score on a particular variable be taken? The answer is that such scales can be conceived of most usefully as "gap detectors," indicating which aspects of policy are proceeding well and which are problematic. They do not, however, reveal why the dysfunctions have developed. nor do they suggest what must be done to correct them.

Policy Formation

Evaluating nutrition policy at its earliest. conceptual stage basically involves identifying which agent (or agents) were responsible for proposing and developing a policy, what objectives the proposers were attempting to achieve, and the degree of support or opposition they encountered in the process. Careful analysis of the policy formation stage should reveal the points at which the final prescription diverged from the original conception and/or where the approach adopted was, in restrospect, ill-conceived.

Origin of Policy

The more government has been involved in initiating and formulating a policy, the more committed it is to that policy and its effective implementation. International agencies often dominate the process for several reasons: (a) because they assume they know what needs to be done, (b) because they have the technical expertise to design policies (and later) programmes, and (c) because they wish to further their own objectives. Yet, policies have proven to be more effective when they originate with a broad-based contingent of powerful officials within the country who can influence both the shape and the implementation of the public agenda.

Performance Scale

0 - Exclusively externally initiated/No local input.
1 - Predominantly externally initiated/Very little local input.
2 - Strong external initiative/Minimal local input.
3 - Some internal initiative with much external influence.
4 - Strong internal initiative with moderate external influence.
5 - Primarily internal initiative with external aid as required.

Motivation for Policy

In order for a policy to be successful, it is necessary for there to be a strong commitment to social development, which might be measured in one respect in terms of orientation to "basic needs" programming. Thus, an evaluator can gauge a country's commitment to nutrition policy by the extent of attention to such things as rural health care delivery, housing, and primary education.

Performance Scale

0 - Symbolic social programming ("palliative" approach).
1 - Primarily politically motivated/Little interest in social development.
2 - Politically motivated/Some interest in social development.
3 - Moderate interest in social development.
4 - Substantial concern for social development.
5 - Strong concern for and commitment to social development.

Support for Policy

For a nutrition policy to be effective, it must enjoy widespread support at all levels of government. Narrowlybased support emanating from one or two "charismatic individuals" may succeed in getting a new policy approach off the ground, but it can be reversed quickly. The institutionalization of the support, therefore, is often manifested in a coordinating body (e.g., a nutrition council) that is capable of insuring that requisite actions are carried out by the involved government agencies.

Performance Scale

0 - No support/Externally motivated.
1 - Very narrow support/Dependent on a single "charismatic" individual.
2 - Narrow support/Limited to several actors in one ministry.
3 - Several ministries involved but no formal coordinating body.
4 - Formal coordinating group exists but has little or no power.
5 - Formal coordinating body exists with authority to take action.

Special Interests

The role of special interests in the policy process can range from highly negative to positive. The negative aspect may occur either through acts of commission (active opposition) or omission (failure to support). Professional associations, for example, may lobby or pressure against a programme. In contrast, other actors, while not actively opposing a policy, may not provide the support necessary to permit the achievement of objectives. It is also possible for positive and negative special interests to counterbalance each other.

Performance Scale

0 - Special interests have a strongly negative influence.
1 - Special interests have a mildly negative influence.
2 - Special interests have neutral influence.
3 - Special interests have a mildly positive influence.
4 - Special interests have generally positive influence.
5 - Special interests have strongly positive influence.

Objectives of Policy

Experience demonstrates that effectiveness is increased when objectives are clearly stated and goals/targets are established. This facilitates the practice of management by objective, the identification of priorities, and the establishment of a basis for evaluation and accountability.

Performance Scale

0 - No data/Objectives not stated.
1 - Objectives very vague and general.
2 - Objectives specified but not supported by data.
3 - Objectives generally specified and supported by data.
4 - Clearly specified objectives and data/No goals or targets.
5 - Clearly specified objectives with goals/Targets corresponding to priorities indicated by data.

Programme Development

Evaluating nutrition policy at the level of programme development involves primarily the assessment of the various mechanisms established or employed to translate goal statements into operational programmes. It requires a thorough understanding of both the political and bureaucratic processes as well as the role played (if any) by external agencies as initiators and facilitators.

Policy-Programme Linkages

Policy must be translated into action by means of programmes. Most often an effective nutrition policy will require a number of programmes in order to achieve its objectives. Potential success, therefore, can be identified by determining how closely programmes fulfill the objectives outlined in a policy.

Performance Scale

0 - No programme/No relationship to policy.
1 - Little relationship to policy.
2 - Some relationship to policy.
3 - Close relationship to policy.
4 - Strong relationship to policy.
5 - Fully congruent with policy.

Programme Design

Programmes based on communications with those for whom the benefits are intended and on genuinely perceived felt needs of the target group are more likely to succeed. The centrally planned, top-down approach attempts to apply a limited programmatic conception to large areas with widely varying conditions and problems. Moreover, participation in determining programme content increases target group interest and support. Consultation serves the dual purpose of exposing target group concerns and problems as well as informing people of the availability of new programme services.

Performance Scale

0 - Total lack of awareness of target group needs.
1 - Limited knowledge of target group needs.
2 - Moderate knowledge of target group needs.
3 - Extensive knowledge of target group needs/Limited consultation.
4 - Consultation with target group/But not reflected in programme design.
5 - Extensive consultation with target group resulting in response to felt needs.

Action Agencies

If the lead or "action" ministry in the nutrition programming effort is to carry out the policy with the necessary degree of commitment, it must have an interest in the problem. Assigning nutrition activities to an organization or department that has neither the interest, the knowledge, nor the experience in nutritionrelated programmes (e.g., an agriculture ministry) decreases the potential for impact.

Performance Scale

0 - "Assigned" to agency involuntarily.
1 - Directed by higher authority to undertake/Little relevance to mission.
2 - Directed to undertake/Some interest and relevance to mission.
3 - Agency already active in the field/Limited interest.
4 - Agency active and interested/Pursuit of issue is voluntary.
5 - Voluntary leadership/Pursuit of issues with enthusiasm.

Awareness of Management Issues

To implement nutrition intervention programmes successfully, programme designers must have a deep awareness of the ingredients of successful programming. In this respect, the "form" of the intervention (e.g., supplementary feeding programme, etc.) will only provide superficial guidelines as to how the programme is to be designed and carried out. The organizational and management-related factors necessary to attain programme objectives must also be considered. Among these concerns are: personnel. training, information systems, supervision, and structural constraints.

Performance Scale

0 - No awareness or exposure to organization and management (O & M) issues.
1 - Little awareness or exposure to O & M issues.
2 - Some awareness and exposure to O & M issues.
3 - Significant awareness and exposure to O & M issues.
4 - Process studies conducted/Extensive awareness.
5 - Process studies conducted/Complete awareness.

Budget

Budgetary allocations for nutrition programmes are often taken as a measure of a government's commitment to its stated policy. However, it is difficult to ascertain exactly what budgetary support means in absolute terms. More revealing as a manifestation of government commitment is the extent of funding for preventive versus curative programmes. While ratios may not be totally accurate indicators (since curative care costs more than preventive), the figures provide some sense of where the priorities of the government are focused.

Performance Scales

a) Ratio of rural to urban per capita expenditure on health:
0 - 1:10.
1 - Between 1:10and 1:5.
2 - Between 1:5 and 1:3.
3 - Between 1 :3 and 1 :2.
4 - Between 1:2 and 1:1.
5 - Better than 1:1.

b) Nutrition programmes as a percentage of the total national budget:
0 - Less than 1 per cent.
1 - 1-3 per cent.
2 - 3-5 per cent.
3 - 5-8 per cent.
4 - 8-10 per cent.
5 - Greater than 10 per cent.

Programme Implementation

The third level of nutrition policy evaluation concerns the broad range of implementation activities undertaken to deliver services to the intended target populations. It is frequently at this level, in fact, that nutrition policies break down; that is, policy is formulated and programmes are established but, for a variety of reasons, the planned services are not delivered as originally intended. There are many causes of this system breakdown; they relate mainly to issues of logistics, politics, and economics. Moreover, evaluators must also consider the stage (or age) of programme development, because this is likely to alter the manner in which programme elements fit together and are evaluated.

Infrastructure/Basic Supplies

Little can be expected of a programme that is not supported by the necessary infrastructure (e.g., buildings, staff, supplies, transportation networks, etc.). Deficiencies in professional and pare-professional training and availability, lack of adequate facilities, and shortages of medicine, food and equipment can prevent competent, well-trained workers from implementing a programme as planned.

Performance Scale

0 - Necessary infrastructure non-existent.
1 - Most infrastructure unavailable or under development.
2 - Some infrastructure available/Operational effectiveness doubtful.
3 - Some infrastructure operational/Nothing more under development.
4 - Most infrastructure operational/Remainder under or nearing completion.
5 - All necessary infrastructure available and operational.

Programme Decentralization

Experience demonstrates that most programmes (with the exception of such technical schemes as fortification) operate more effectively if administered within a decentralized framework. This arrangement permits the adaptation of programme interventions to local conditions and circumstances, and it allows for the flexibility to respond to unique demands and situations. This contrasts sharply with the rigid, highly centralized bureaucracy that will not permit any modifications and, hence, loses substantial amounts of potential impact.

Performance Scale

0 - Total centralization.
1 - Some decentralization exists/Little effective impact.
2 - Moderate decentralization/Continued heavy reliance on central government.
3 - Substantial decentralization/Very limited local autonomy.
4 - Substantial decentralization/Substantial autonomy.
5 - Effective decentralization with virtually total autonomy.

Self-Reliance

Successful nutrition schemes must be institutionalized. This is particularly important in programmes that require behavioural change (e.g., nutrition education). Little in the way of long-term results can be expected without the establishment of self-reliant momentum. Indicators of self-reliance include the extent of local support for a programme, the degree of local organization (institutionalization), and the level of local involvement in programme design, implementation and evaluation. More top-down, dependency-creating approaches tend to foster welfare ("let them do it for me") attitudes that reduce likely policy impact.

Performance Scale

0 - Paternalistic/Perpetuates dependency.
1 - Self-reliance espoused/Dependency maintained.
2 - Nominal self-reliance.
3 - Some self-reliance attained.
4 - Significant self-reliance attained.
5 - Total self-reliance attained.

Political Elite Support

Political elites are not only important in the formation of policy but are also vital in the implementation of programmes as well. National and local elected officials can, by actively opposing a programme, decrease dramatically the chances of a policy's achieving results. Similar negative effects result from weak, ineffective support by political elites. It is highly unlikely that a nutrition programme can succeed at the local level without the active support of local political and opinion leaders.

Performance Scale

0 - Political elite actively opposed.
1 - Political elite moderately opposed.
2 - Political elite neutral.
3 - Political elite moderately supportive.
4 - Political elite actively supportive.
5 - Political elite acting as the initiating agent.

Results Orientation

Management by objective is concerned with results as distinguished from other types of management which are concerned chiefly with procedure. Management by objective may be conceptualized as follows:

The more results-oriented the programme implementation, the greater the chance for programme impact. While not all the characteristics may be found in a particular programme, the mix of interventions should have the applicable characteristics if the policy is to be judged a success.

Performance Scale

0 - Completely procedure oriented.
1 - Results-oriented in word/Still largely procedureoriented.
2 - Mixed results and procedure orientations.
3 - Modestly results-oriented.
4 - Substantially resultsoriented.
5 - Completely results-oriented.

Targeting

Although impact is the ultimate measure of successful programme implementation, an intermediate indicator of the potential effectiveness of a targeted programme is the ability to reach those for whom the scheme was designed. Delivery of services to the most seriously disadvantaged is notoriously difficult, especially in hardto-reach rural areas. Special efforts must be made to include the politically powerless and unorganized.

Performance Scale

0 - No evidence of successful service delivery.
1 - Services reaching less than 5 per cent of target group.
2 - Services reaching between 5-25 per cent of target group.
3 - Services reaching 25-50 per cent of target group.
4 - Services reaching 50-75 per cent of target group.
5 - Services effectively reaching more than 75 per cent of target group.