A policy evaluation methodology
Two basic assumptions underlie the methodology described here. First, it is
assumed that the policy evaluations are to be conducted either retrospectively
or concurrent with operational programmes, i.e., that the focus is on the
origins and ramifications of past and present policies. A second, related,
assumption is that the policy evaluations are not intended to be prospective,
i.e., that the principal objective is not necessarily the development of new
programmatic approaches (although this is certainly one possible outcome)
Because the conditions under which policies evolve and are implemented vary
substantially, depending on the country context, it is neither possible nor
advisable to spell out a "cookbook" approach that can be applied
without modification under all circumstances. Rather, we have outlined here a
generic methodology that can (and should) be modified and adapted to the
specific type of nutrition policy being evaluated.
Our approach has been to set out a series of scales for each of the three
policy stages by means of which performance factors can be measured and the
impact of a given policy evaluated. When employed successfully, the overall
effectiveness of a country's nutrition policy may be determined by comparing
actual performance with the potential score. Of course, there is also a problem
with the use of scales in attempting to weight the importance of one parameter
relative to another; that is, how seriously should a low score on a particular
variable be taken? The answer is that such scales can be conceived of most
usefully as "gap detectors," indicating which aspects of policy are
proceeding well and which are problematic. They do not, however, reveal why the
dysfunctions have developed. nor do they suggest what must be done to correct
Evaluating nutrition policy at its earliest. conceptual stage basically
involves identifying which agent (or agents) were responsible for proposing and
developing a policy, what objectives the proposers were attempting to achieve,
and the degree of support or opposition they encountered in the process. Careful
analysis of the policy formation stage should reveal the points at which the
final prescription diverged from the original conception and/or where the
approach adopted was, in restrospect, ill-conceived.
Origin of Policy
The more government has been involved in initiating and formulating a policy,
the more committed it is to that policy and its effective implementation.
International agencies often dominate the process for several reasons: (a)
because they assume they know what needs to be done, (b) because they have the
technical expertise to design policies (and later) programmes, and (c) because
they wish to further their own objectives. Yet, policies have proven to be more
effective when they originate with a broad-based contingent of powerful
officials within the country who can influence both the shape and the
implementation of the public agenda.
0 - Exclusively externally initiated/No local input.
1 - Predominantly
externally initiated/Very little local input.
2 - Strong external
initiative/Minimal local input.
3 - Some internal initiative with much
4 - Strong internal initiative with moderate external
5 - Primarily internal initiative with external aid as required.
Motivation for Policy
In order for a policy to be successful, it is necessary for there to be a
strong commitment to social development, which might be measured in one respect
in terms of orientation to "basic needs" programming. Thus, an
evaluator can gauge a country's commitment to nutrition policy by the extent of
attention to such things as rural health care delivery, housing, and primary
0 - Symbolic social programming ("palliative" approach).
Primarily politically motivated/Little interest in social development.
Politically motivated/Some interest in social development.
3 - Moderate
interest in social development.
4 - Substantial concern for social
5 - Strong concern for and commitment to social development.
Support for Policy
For a nutrition policy to be effective, it must enjoy widespread support at
all levels of government. Narrowlybased support emanating from one or two
"charismatic individuals" may succeed in getting a new policy approach
off the ground, but it can be reversed quickly. The institutionalization of the
support, therefore, is often manifested in a coordinating body (e.g., a
nutrition council) that is capable of insuring that requisite actions are
carried out by the involved government agencies.
0 - No support/Externally motivated.
1 - Very narrow support/Dependent on
a single "charismatic" individual.
2 - Narrow support/Limited to
several actors in one ministry.
3 - Several ministries involved but no
formal coordinating body.
4 - Formal coordinating group exists but has
little or no power.
5 - Formal coordinating body exists with authority to
The role of special interests in the policy process can range from highly
negative to positive. The negative aspect may occur either through acts of
commission (active opposition) or omission (failure to support). Professional
associations, for example, may lobby or pressure against a programme. In
contrast, other actors, while not actively opposing a policy, may not provide
the support necessary to permit the achievement of objectives. It is also
possible for positive and negative special interests to counterbalance each
0 - Special interests have a strongly negative influence.
1 - Special
interests have a mildly negative influence.
2 - Special interests have
3 - Special interests have a mildly positive influence.
4 - Special interests have generally positive influence.
5 - Special
interests have strongly positive influence.
Objectives of Policy
Experience demonstrates that effectiveness is increased when objectives are
clearly stated and goals/targets are established. This facilitates the practice
of management by objective, the identification of priorities, and the
establishment of a basis for evaluation and accountability.
0 - No data/Objectives not stated.
1 - Objectives very vague and
2 - Objectives specified but not supported by data.
Objectives generally specified and supported by data.
4 - Clearly specified
objectives and data/No goals or targets.
5 - Clearly specified objectives
with goals/Targets corresponding to priorities indicated by data.
Evaluating nutrition policy at the level of programme development involves
primarily the assessment of the various mechanisms established or employed to
translate goal statements into operational programmes. It requires a thorough
understanding of both the political and bureaucratic processes as well as the
role played (if any) by external agencies as initiators and facilitators.
Policy must be translated into action by means of programmes. Most often an
effective nutrition policy will require a number of programmes in order to
achieve its objectives. Potential success, therefore, can be identified by
determining how closely programmes fulfill the objectives outlined in a policy.
0 - No programme/No relationship to policy.
1 - Little relationship to
2 - Some relationship to policy.
3 - Close relationship to
4 - Strong relationship to policy.
5 - Fully congruent with
Programmes based on communications with those for whom the benefits are
intended and on genuinely perceived felt needs of the target group are more
likely to succeed. The centrally planned, top-down approach attempts to apply a
limited programmatic conception to large areas with widely varying conditions
and problems. Moreover, participation in determining programme content increases
target group interest and support. Consultation serves the dual purpose of
exposing target group concerns and problems as well as informing people of the
availability of new programme services.
0 - Total lack of awareness of target group needs.
1 - Limited knowledge
of target group needs.
2 - Moderate knowledge of target group needs.
Extensive knowledge of target group needs/Limited consultation.
Consultation with target group/But not reflected in programme design.
Extensive consultation with target group resulting in response to felt needs.
If the lead or "action" ministry in the nutrition programming
effort is to carry out the policy with the necessary degree of commitment, it
must have an interest in the problem. Assigning nutrition activities to an
organization or department that has neither the interest, the knowledge, nor the
experience in nutritionrelated programmes (e.g., an agriculture ministry)
decreases the potential for impact.
0 - "Assigned" to agency involuntarily.
1 - Directed by higher
authority to undertake/Little relevance to mission.
2 - Directed to
undertake/Some interest and relevance to mission.
3 - Agency already active
in the field/Limited interest.
4 - Agency active and interested/Pursuit of
issue is voluntary.
5 - Voluntary leadership/Pursuit of issues with
Awareness of Management Issues
To implement nutrition intervention programmes successfully, programme
designers must have a deep awareness of the ingredients of successful
programming. In this respect, the "form" of the intervention (e.g.,
supplementary feeding programme, etc.) will only provide superficial guidelines
as to how the programme is to be designed and carried out. The organizational
and management-related factors necessary to attain programme objectives must
also be considered. Among these concerns are: personnel. training, information
systems, supervision, and structural constraints.
0 - No awareness or exposure to organization and management (O & M)
1 - Little awareness or exposure to O & M issues.
2 - Some
awareness and exposure to O & M issues.
3 - Significant awareness and
exposure to O & M issues.
4 - Process studies conducted/Extensive
5 - Process studies conducted/Complete awareness.
Budgetary allocations for nutrition programmes are often taken as a measure
of a government's commitment to its stated policy. However, it is difficult to
ascertain exactly what budgetary support means in absolute terms. More revealing
as a manifestation of government commitment is the extent of funding for
preventive versus curative programmes. While ratios may not be totally accurate
indicators (since curative care costs more than preventive), the figures provide
some sense of where the priorities of the government are focused.
a) Ratio of rural to urban per capita expenditure on health:
0 - 1:10.
1 - Between 1:10and 1:5.
2 - Between 1:5 and 1:3.
3 - Between 1
:3 and 1 :2.
4 - Between 1:2 and 1:1.
5 - Better than 1:1.
b) Nutrition programmes as a percentage of the total national budget:
- Less than 1 per cent.
1 - 1-3 per cent.
2 - 3-5 per cent.
5-8 per cent.
4 - 8-10 per cent.
5 - Greater than 10 per cent.
The third level of nutrition policy evaluation concerns the broad range of
implementation activities undertaken to deliver services to the intended target
populations. It is frequently at this level, in fact, that nutrition policies
break down; that is, policy is formulated and programmes are established but,
for a variety of reasons, the planned services are not delivered as originally
intended. There are many causes of this system breakdown; they relate mainly to
issues of logistics, politics, and economics. Moreover, evaluators must also
consider the stage (or age) of programme development, because this is likely to
alter the manner in which programme elements fit together and are evaluated.
Little can be expected of a programme that is not supported by the necessary
infrastructure (e.g., buildings, staff, supplies, transportation networks,
etc.). Deficiencies in professional and pare-professional training and
availability, lack of adequate facilities, and shortages of medicine, food and
equipment can prevent competent, well-trained workers from implementing a
programme as planned.
0 - Necessary infrastructure non-existent.
1 - Most infrastructure
unavailable or under development.
2 - Some infrastructure
available/Operational effectiveness doubtful.
3 - Some infrastructure
operational/Nothing more under development.
4 - Most infrastructure
operational/Remainder under or nearing completion.
5 - All necessary
infrastructure available and operational.
Experience demonstrates that most programmes (with the exception of such
technical schemes as fortification) operate more effectively if administered
within a decentralized framework. This arrangement permits the adaptation of
programme interventions to local conditions and circumstances, and it allows for
the flexibility to respond to unique demands and situations. This contrasts
sharply with the rigid, highly centralized bureaucracy that will not permit any
modifications and, hence, loses substantial amounts of potential impact.
0 - Total centralization.
1 - Some decentralization exists/Little
2 - Moderate decentralization/Continued heavy reliance on
3 - Substantial decentralization/Very limited local
4 - Substantial decentralization/Substantial autonomy.
Effective decentralization with virtually total autonomy.
Successful nutrition schemes must be institutionalized. This is particularly
important in programmes that require behavioural change (e.g., nutrition
education). Little in the way of long-term results can be expected without the
establishment of self-reliant momentum. Indicators of self-reliance include the
extent of local support for a programme, the degree of local organization
(institutionalization), and the level of local involvement in programme design,
implementation and evaluation. More top-down, dependency-creating approaches
tend to foster welfare ("let them do it for me") attitudes that reduce
likely policy impact.
0 - Paternalistic/Perpetuates dependency.
1 - Self-reliance
2 - Nominal self-reliance.
3 - Some
4 - Significant self-reliance attained.
Total self-reliance attained.
Political Elite Support
Political elites are not only important in the formation of policy but are
also vital in the implementation of programmes as well. National and local
elected officials can, by actively opposing a programme, decrease dramatically
the chances of a policy's achieving results. Similar negative effects result
from weak, ineffective support by political elites. It is highly unlikely that a
nutrition programme can succeed at the local level without the active support of
local political and opinion leaders.
0 - Political elite actively opposed.
1 - Political elite moderately
2 - Political elite neutral.
3 - Political elite moderately
4 - Political elite actively supportive.
5 - Political elite
acting as the initiating agent.
Management by objective is concerned with results as distinguished from other
types of management which are concerned chiefly with procedure. Management by
objective may be conceptualized as follows:
| ||Results ||Procedure |
|Objectives ||Specific ||Vague |
|Target Groups ||Priorities ||All |
|Indicators of Success ||Outputs
|Organization ||Team ||Specialists |
|Training ||Problem-Solving ||Routines |
The more results-oriented the programme implementation, the greater the
chance for programme impact. While not all the characteristics may be found in a
particular programme, the mix of interventions should have the applicable
characteristics if the policy is to be judged a success.
0 - Completely procedure oriented.
1 - Results-oriented in word/Still
2 - Mixed results and procedure orientations.
3 - Modestly results-oriented.
4 - Substantially resultsoriented.
5 - Completely results-oriented.
Although impact is the ultimate measure of successful programme
implementation, an intermediate indicator of the potential effectiveness of a
targeted programme is the ability to reach those for whom the scheme was
designed. Delivery of services to the most seriously disadvantaged is
notoriously difficult, especially in hardto-reach rural areas. Special efforts
must be made to include the politically powerless and unorganized.
0 - No evidence of successful service delivery.
1 - Services reaching
less than 5 per cent of target group.
2 - Services reaching between 5-25 per
cent of target group.
3 - Services reaching 25-50 per cent of target
4 - Services reaching 50-75 per cent of target group.
Services effectively reaching more than 75 per cent of target