Cover Image
close this bookHealth Economics for Developing Countries: A Survival Kit (London School of Hygiene and Tropical Medicine, 1998, 134 p.)
View the document(introduction...)
View the documentPublication Series - Health Policy Unit
View the documentAcknowledgements
View the documentPreface
View the documentChapter 1: Health Economics and its Contribution to Health Planning
View the documentChapter 2: Economic Development and Health
View the documentChapter 3: Financing Economic and Health Development
View the documentChapter 4: Health Care: the State versus the Market
View the documentChapter 5: Demand, Supply and the Price System
View the documentChapter 6: Concepts of Economic Efficiency
View the documentChapter 7: Inputs, Resources and Costs
View the documentChapter 8: Outputs, Health and Health Indicators
View the documentChapter 9: The Techniques of Economic Evaluation
View the documentChapter 10: National Accounts and the Health Sector
View the documentChapter 11: Health Sector Finance and Expenditure
View the documentChapter 12: Sources of Finance for the Health Sector
View the documentChapter 13: Budgetary Procedures: Budgetary Reform and Programme Budgeting
View the documentChapter 14: Approaches to Financial Planning: Resource Allocation Planning and the Financial Master Plan
View the documentSelected Bibliography
View the documentGlossary
View the documentBack Cover

Chapter 9: The Techniques of Economic Evaluation

1. The Definition and Forms of Economic Evaluation

Economic evaluation studies generally seek to address the two questions posed in Chapter 6: Are limited resources used in the best ways possible? Is value for money achieved in their use? The techniques of economic evaluation are, therefore, an important contribution to the methods of health service evaluation and are based on the economist's concern with economic efficiency and opportunity costs.

Economic evaluation has been defined as:

“the quantitative analysis of the relative desirability to the whole community of investing in alternative projects or programmes”,

where desirability is assessed in terms of both costs and consequences. 'Consequences' is used here as the generic term for the beneficial results of a programme (often termed effects or benefits, depending on the techniques of analysis being used). Within this broad definition there are many forms of economic evaluation, as shown in Figure 18. Only those forms which examine both costs and consequences for two or more alternatives fit the above definition and can be described as full economic evaluation studies. In practice, one of the two alternatives examined may be an existing project/programme (the 'do-nothing' alternative).

Cost-minimisation analysis (CMA) is based on prior epidemiological findings which show that the outcome of interest (e.g. reduction of disability) is achieved to the same degree by two (or more) interventions. The technique is used to identify the least cost intervention.

Cost-effectiveness analysis (CEA) investigates the best way of achieving a single objective by comparing effects and costs. It evaluates either:

- which of a number of possible interventions will achieve a given health objective at least cost, or

- given a fixed budget, the intervention that maximises the effectiveness of the expenditure.

Its results are expressed either as costs per unit of output (total costs of the intervention divided by total health effect) or as effect per monetary unit (total health effect divided by total available resources).

Figure 18: Forms of Economic Evaluation

Are both costs and consequences examined?



No

Yes



Examines consequences only

Examines costs only

Cost-outcome description

Is there a comparison of

No

Outcome description

Cost description


2 or more alternatives?

Yes

Effectiveness evaluation

Cost analysis

1. Cost minimization analysis
2. Cost effectiveness analysis
3. Cost benefit analysis
4. Cost utility analysis

Adapted from: McMaster University Health Sciences Centre (1984). How to read clinical journals: VII. To understand an economic evaluation (part A) Canadian Medical Association Journal, Vol 130, June 1, p. 1430.

Cost-benefit analysis (CBA) values both costs and benefits in monetary terms, and compares them, assessing whether the project/programme is desirable through the use of decision criteria (e.g. if the benefit cost ratio (benefits divided by costs) is greater than one, the project/programme is worthwhile).

Cost-utility analysis (CUA) is a form of CEA but it measures the effects of a project/programme in terms of utilities (the quality-adjusted health outcome caused or averted). Like CEA it can focus on either minimizing cost or maximizing effect; and its results are expressed, for example, in terms of costs per Quality Adjusted Life Year (QALY) or QALYs per monetary unit.

The root differences between these techniques concern their evaluation of health outcome (consequence) and their breadth of analysis. CMA and CEA tacitly assume that the health objectives which the projects/programmes serve are worthwhile. CUA permits choice between a much wider range of interventions but still ultimately assumes that at some cut-off point of cost per QALY, a programme is worthwhile. CBA in theory permits assessment of whether the health objectives are worth achieving in the first place.

2. The Steps of Economic Evaluation

All economic evaluation techniques involve three basic steps:

- identification of costs and consequences
- measurement of costs and consequences
- valuation of costs or of costs and consequences.

In addition, all economic evaluation studies should consider adjusting costs and consequences for differential timing, and should incorporate an incremental and a sensitivity analysis.

2.1 Identification of costs and consequences

Table 2 establishes the types of costs and consequences relevant to economic evaluation in the health sector. Although it may be impossible to measure and value all of these items it is important that they are, at least, identified, in order to clarify any possible analytical bias in favour of the most easily measurable items.

It may appear obvious that costs and consequences for both the health sector and for patients should he considered, but many studies confine their attention to the health sector only. This approach limits analysis and is contrary to the definition of economic evaluation, which stresses that the relative desirability of alternatives to the whole community should be the focus of evaluation. It can bias study results against certain population groups or interventions.

Table 2: Types of costs and consequences relevant to the economic evaluation of health care projects/programmes

Costs


1.

Organizing and operating costs within the health care sector (e.g. health care professionals' time, supplies, equipment, power and capital costs)

}
}


2.

Costs borne by patients and their families:

} direct



out-of-pocket expenses

} costs



patient and family inputs into treatment

}



time lost from work

} indirect




psychic costs

} costs



3.

Costs borne externally to the health care sector, patients and their families


Consequences


1.

Changes in physical, social or emotional functioning (effects)



2.

Changes in resources use (benefits)




for organizing and operating services within the health care sector:





- for the original condition

} direct




- for unrelated conditions

} benefits



relating to activities of patients and their families:





- savings in expenditure or leisure time

} direct benefits




- savings in lost work time

} indirect benefits


3.

Changes in the quality of life of patients and their families (utility)

Source: Drummond MF and Stoddart GL (1985) Principles of Economic Evaluation of Health Programmes World Health Statistics Quarterly 38(4): p 360

For example, consider an evaluation of immunization services in urban and rural areas. By focusing on health sector costs and consequences and including 'cost savings to the health sector', the evaluation would probably favour the concentration of immunization services in towns. Health services are more developed in urban than in rural areas and so the potential savings to the health sector from preventing illness are greater. However, in under-provided rural areas, the local provision of immunization services would generate savings for the patients in the form of lower transport costs in obtaining treatment and less private treatment and self-care. These should be considered within the evaluation and may lessen its bias towards urban populations. It is the concern to minimize government (rather than social) costs that often determines the conduct of economic evaluation studies and may be used (wrongly) to justify an exclusive focus on health sector costs/consequences.

The third cost item, costs outside the health sector, is much less obvious and may not be relevant to all studies. Its inclusion stresses the need to adopt a community-wide perspective to clarify the full implication of any project/programme. For example, an industrial safety measure may alter the cost or availability of products if it results in more costly production processes.

It is sometimes difficult to distinguish between the costs of an intervention and the changes in resource use that result from it: Which is the cost and which the consequence? The clearest way of seeing the difference is to remember that the consequences in question are often termed 'benefits', that is they reflect some positive change as a result of the intervention. If a result of an intervention is time saved (less time lost from work) it is, therefore, a benefit; but if time costs are increased, perhaps because of lengthy treatment (more time lost from work), the increase is a cost.

The differences between the consequences identified in the table help to distinguish the differences between CEA, focusing on effects, CBA, focusing on benefits, and CUA, focusing on utilities.

2.2 Measurement of costs and consequences

The measurement of costs and consequences in developing countries is undermined by the lack of routine statistical data. Expenditure data may be considerably out of dale and even when it is available is unlikely to be fully accurate or broken down into appropriate expenditure categories (e.g. disaggregated by institution).

It is better to base project/programme costing on detailed consideration of the actual physical inputs used. This would also allow appropriate division of the costs shared by many services; for example, the World Health Organization Expanded Programme of Immunization costing manual recommends that the immunization share of such joint costs (for instance supervision and administration) be determined on the basis of staff time allocations to immunization relative to the other services sharing the resources. However, identification of all the physical inputs used by a programme could be a costly process in itself. It may be most appropriate to focus such detailed consideration on those inputs likely to represent the major portion of costs and/or which can be used to split joint costs (e.g. staff lime).

Measurement of consequences in terms of health status effects is even more difficult, given the lack of routine epidemiological data in developing countries and the weaknesses of the epidemiological evidence concerning the links between health care inputs and health status outcomes. As a result of the many influences on health it is extremely difficult to disentangle the health impact of a health intervention from the influence of other factors (such as improved education, better water supplies etc.). Moreover, a service delivery package may be too complex to allow appropriate measurement of effects.

For CEA, the problem is made worse by the need to choose an appropriate single unit of effect. Chapter 8 outlined the measures of health effect that may be used, and stressed that because health is multi-dimensional it is difficult to measure it along one dimension only. For example, the unit of 'death prevented' cannot measure the impact on disability that may be an important health consequence. It is also only projects/programmes that have the same objective in terms of health status improvement whose consequences can be measured using the same unit of effect. 'Deaths prevented' cannot be used to compare, for example, a malaria control programme which prevents many infant deaths and many episodes of acute illness, and a schistosomiasis control programme which prevents widespread chronic disability and relatively few adult deaths.

Using a single unit of effect necessarily ignores the non health status consequences of programmes, such as the time-saving effects of water supplies. Ignoring these wider consequences undervalues those interventions such as water supply programmes which have these broader effects. In general, primary health care programmes have broad objectives and potential effects that cannot be measured by a single unit of effect, including self-reliance and increased community participation.

Two approaches have been developed to address some aspects of this problem within CEA. The first involves the use of global health indicators and, in particular, quality adjusted life years and healthy days of life lost (sec Chapter 8). These measures focus on changes in the quality of life of patients and their families, permitting consideration of a spectrum of health effects, and are used in cost utility analysis. Data collection for both is very complex and, in practice, QALYs have never been used in an economic evaluation study based in developing countries. They require sophisticated techniques for assessing and measuring community preferences for the different qualities of life associated with different slates of illness, and seem less relevant to the largely acute diseases generally found in developing countries. Measurement of 'healthy days of life lost' requires relevant epidemiological data and most developing country health information systems cannot yet provide them in sufficient detail. This measure also introduces a bias towards interventions that tackle child health problems as children have most healthy days of life to lose, unless some weighting system is introduced.

The other approach that has been adopted within CEA to overcome the problem of choosing an appropriate unit of effect, is the use of units of intermediate output (e.g. numbers of immunized children, proportion of pregnant women attending an antenatal clinic). These are measures of access, coverage or compliance. Where such units are used it is clearly not changes in physical, social or emotional functioning that are evaluated but rather the success of projects/programmes in extending service provision. The evaluation does not assess whether the specific service is worthwhile but rather seeks to identify the best way of providing a service that has previously been identified as worthwhile. The approach may seem like a weak version of economic evaluation as earlier defined, but it is technically more powerful. It is much less open to criticism about the assumptions embodied in the evaluation (related to identification of the project/programme's priority health effect and to the nature of health effects). The link between the input of a health service and the output of increased coverage may be much stronger than the link between a health programme and health status changes. It also provides a useful focus on the means of service delivery and on operational efficiency. However, the appropriateness of the approach clearly depends on the study objectives.

2.3 Valuation of costs and consequences

Chapter 7 introduced the concept of opportunity cost as the economic definition of the sacrifice necessary to obtain a good or service. Although monetary prices may reflect this opportunity cost, there are instances where there is no monetary price or where this price does not accurately reflect the value of the alternative productive use of resources (the economic price). The prices may also not reflect the social costs of activities (the social price). To allow for these problems economists use the techniques of shadow pricing in order to value costs appropriately.

For example, many health projects/programmes use imported goods (e.g. drugs) that are purchased in international markets using foreign exchange. However, the purchase price of these goods will not reflect the opportunity cost of the resources (foreign exchange) used for their purchase where countries have over-valued exchange rates and foreign exchange shortages. Many developing countries suffer this problem and their purchase of drugs with scarce foreign exchange will prevent their purchase of other products that must be bought internationally. Drug costs must, therefore, be valued using a 'shadow foreign exchange rate' to reflect the true opportunity cost of the purchase - raising their value above the actual purchase price.

Alternatively, distortions within the domestic economy, such as tariffs, subsidies, import and export taxes, may shelter it from international competition and exaggerate the opportunity cost of goods produced domestically. It might be possible to purchase the same goods from abroad at lower real cost by using domestic resources to produce exports and exchanging them for the foreign products. World prices may be a better reflection of opportunity costs than domestic prices and can be used to shadow price the domestic production. Non-traded goods can often be valued roughly at equivalent world prices using a standard conversion factor.

Shadow pricing can also be used to establish social prices, allowing for a country's preference for savings versus consumption and/or for income re-distribution. In either case, weights are used to reflect the national preference and, in effect, to bias the choice of projects towards it (e.g. towards the reduction of income inequality, by over-valuing the benefits gained by low income groups, and under-valuing those of high income groups).

Valuing the consequences of health projects/programmes is even more complex than valuing costs. Although CBA does not suffer from the problems of CEA in choosing an appropriate unit of effect, it does suffer from the problems of valuing a diverse range of consequences that includes changes in health status, savings in public and private expenditure on treatment, and economic returns from the exploitation of natural resources (especially land) permitted by programmes such as malaria control. The greatest problem is the valuation of health status improvements in monetary terms. Two main approaches are generally suggested for these valuations: the human capital approach and the willingness to pay approach.

The first is based on the assumption that health projects/programmes represent an investment in people which enables them to be more productive and to increase their material well-being. The benefits to be considered, therefore, include an increase in the availability of labour for productive work and an increase in the productivity of labour whilst at work. The value of any increase in the supply of labour should be based on its marginal product, for which the wage rate is often used as a proxy. However, if the health improvements occur in the subsistence agriculture sector where under-employment may exist, the wage rate may overstate the value of additional units of labour. A 'shadow wage' is, therefore, often used. (The same argument applies to valuing unskilled labour employed in the project/programme).

The willingness to pay approach starts from the assumption that the price consumers pay for a good represents its value to them. Therefore, the amount they would be willing to pay for improved health or reduced likelihood of death represents an appropriate valuation of it.

Both approaches are controversial but they are the first steps in the full valuation of health benefits. The first has been the most extensively used in developing countries, mainly because estimates are feasible. In contrast, it is unclear how willingness-to-pay valuations can be obtained in the absence of a market price (as with health care).

2.4 Adjusting costs and consequences for differential timing

Costs and consequences often occur at different times and health projects/programmes may also differ with respect to the point at which consequences occur. Preventive programmes, for example, may have a delayed impact, unlike curative programmes. Allowance for these differences can be made through the technique of discounting which assumes (on a variety of grounds) that people prefer benefits which occur sooner rather than later. Costs and consequences are discounted to present values by use of a discount rate and greater weight is then given to costs and consequences that arise sooner rather than later. In effect, the procedure is the reverse of applying a rate of interest.

2.5 Incremental analysis

Whatever technique of economic evaluation is used it is important to undertake an incremental analysis i.e. to consider marginal costs. As noted in Chapter 7, most economic decisions are not about whether to produce all or nothing, but rather concern changes in the scale of activities. Should an additional vaccine be added to an existing immunization programme? Should the health infrastructure be expanded? In order to make these decisions it is essential that the additional costs be evaluated separately from the total and average costs of the existing programme, and not be assumed to be identical.

An evaluation of the cost-effectiveness of screening for cancer of the colon in the USA showed the importance of incremental analysis. The average cost per case of cancer detected for the advised protocol of six tests was US$2,500 (the total costs of six tests divided by the total number of cases detected). However, the incremental cost per case detected of the sixth test (having done five) was US$47 million (the cost of the sixth test divided by the number of cases it detected), since the cost of the test remained the same but by the sixth test almost all cases of cancer had already been detected.

2.6 Sensitivity analysis

Given the range of assumptions inherent in most economic evaluation studies it is essential to test the sensitivity of the study results to changes in these assumptions. What impact do different valuation of costs have on the result? or different rates of discount? or different benefit valuations? or different timing?

Sensitivity analysis also allows clarification of the level of detail required in a study. Some studies may generate fairly reliable results using rough data (i.e. they are insensitive to changes in the assumptions); others may require more accurate estimation of the variables.

3. The Problems of Economic Evaluation

In addition to the technical problems of economic evaluation, concerning the lack of information and Hie shortcomings of units of effect, other criticisms of the techniques have arisen from the current debate about the nature of primary health care. Selective primary health care strategies provide those services which have been identified as the most cost-effective using crude CEA techniques (e.g. UNICEF's GOBI-FFF). However, the use of CEA in this way has been criticized on the grounds that 'efficient is not sufficient' as the sole criterion for decision-making; and that economic evaluation techniques have been used inappropriately to impose the value judgements of specialists on the community as a whole.

The argument that 'efficient is not sufficient' is based on the problems of choosing a single unit of effect to measure health status and to evaluate complex medical/health care programmes. In order to compare alternative health interventions using cost effectiveness ratios it is important that all the sources of variation in the indicator are considered. Figure 19 sets out the factors that affect this ratio. To assume, for example, on the basis of an experimental study, that the intervention with the lowest cost per death prevented will permit more deaths prevented for the available resources than other interventions is also to assume that the intervention will actually be used by the population. However, utilization is dependent both on acceptability and on accessibility, i.e. on the nature of the delivery strategy, which is often not considered in an experimental situation (for similar reasons it is difficult to transfer CEA findings from one country to another).

For CEA it is also important that the alternatives under examination are similar in all respects. For example, the cost-effectiveness ratio is influenced by the number of people covered by a project/programme and a lower ratio may simply reflect a larger catchment population rather than effectiveness. Thus, where two interventions have the same cost-effectiveness ratio but serve different populations it is not clear which is the most desirable.


Figure 19: Factors affecting the cost-effectiveness ratio

Source: Berman P (1982) Selective Primary Health Care: Is Efficient Sufficient? Social Science and Medicine 16, p 1056.

The steps of economic evaluation discussed in section 2.2 indicate that the generation of even a simple cost effectiveness ratio requires many value judgements - concerning the appropriate objectives of health programmes, the appropriate valuation of costs and consequences, the time preferences of the community etc. Often these judgements are made by specialists with no consideration of community preferences and they are rarely identified clearly in the studies undertaken. In practice, economic evaluation techniques may be used as a black-box into which assumptions are fed and from which the basis of resource use decisions are produced. The resulting decisions may be made without consideration of community views and so in contradiction of one of the basic tenets of primary health care. It is vital that assumptions in studies be clarified, and that study results are not quoted in isolation from their context.

4. The Strengths of Economic Evaluation

Against these criticisms can be set the reality of scarce resources, the need to assess whether value for money is being obtained through their use and the exclusive nature of choice. No person can be in two places at the same time, and any use of resources denies their use elsewhere. It is, therefore, important to obtain the best value for money from the use of scarce resources.

Economic evaluation techniques provide a framework within which to consider these issues. They are adaptable and can begin to account for many concerns - the nature, magnitude and timing of cost and consequence, consideration of community preferences etc.

In particular, they have an important role to play in addressing 'management issues' where the emphasis is on assessing how best to provide services, rather than identification of priorities for investment. For example: Should services be provided in health facilities or within homes? Should they be provided during the daytime or in the evening? Such issues have been under-evaluated but better strategies will promote more efficient resource use.

The choice of technique obviously depends on the objective of the study. Broad evaluations of health consequences or of programmes in different sectors require the use of CBA or CUA. More constrained evaluations, for which a single unit of effect can validly summarize project/programme impact, can be undertaken with CEA. CEA is especially useful in considering strategies for service provision. However, the techniques used must also reflect an assessment of the methodological difficulties of applying them to the particular study and in the particular context.

A decision on when to use the techniques must rest on consideration of whether:

- the volume of resources at slake justifies the study
- there are clear alternative uses for the resources to be evaluated
- the technology underlying each alternative is sufficiently well understood
- a reasonable length of time is available for the study
- decision-makers are receptive to the results of the study and have not already made up their minds.

5. Ten Questions to Ask of any Study

Decision-makers in the health sector may often find themselves in the position of receiving an economic evaluation study on the basis of which they may be expected to take some action. The following questions provide a framework for assessing the results of any evaluation study. They summarize the various issues discussed in this chapter.

1.

Was a well-defined question posed in answerable form?

(a)

Did the study examine both costs and effects of the service(s) or programme(s)?

(b)

Did the study involve a comparison of alternatives?

(c)

Was a viewpoint for the analysis stated or was the study placed in a particular decision-making context?

2.

Was a comprehensive description of the competing alternatives given (i.e. can you tell who did what to whom, where, and how often)?

(a)

Were any important alternatives omitted?

(b)

Was (should) a 'do-nothing' alternative (have been) considered?

3.

Was there evidence that the programmes' effectiveness had been established?


Was this done through a randomized, controlled clinical trial? If not, how strong was the evidence of effectiveness?

4.

Were all important and relevant costs and consequences for each alternative identified?

(a)

Was the range wide enough for the research question at hand?

(b)

Did it cover all relevant viewpoints (e.g. those of the community or society, patients and third-party payers)?

(c)

Were capital costs as well as operating costs included?

5.

Were costs and consequences measured accurately in appropriate physical units (e.g. hours of nursing time, number of physician visits, days lost from work or years of life gained) prior to valuation?

(a)

Were any identified items omitted from measurement? If so, does this mean that they carried no weight in the subsequent analysis?

(b)

Were there any special circumstances (e.g. joint use of resources) that made measurement difficult? Were these circumstances handled appropriately?

6.

Were costs and consequences valued credibly?

(a)

Were the sources of all values (e.g. market values, patient or client preferences and views, policy makers' views and health care professionals' judgements) clearly identified?

(b)

Were market values used for changes involving resources gained or used?

(c)

When market values were absent (e.g. when volunteers were used) or did not reflect actual values (e.g. clinic space was donated at a reduced rate) were adjustments made to approximate market values?

(d)

Was the valuation of consequences appropriate for the question posed (i.e. was the appropriate type, or types, of analysis - cost-effectiveness, cost-benefit or cost-utility - selected)?

7.

Were costs and consequences adjusted for differential timing?

(a)

Were costs and consequences that occurred in the future 'discounted' to their present values?

(b)

Was any justification given for the discount rate used?

8.

Was an incremental analysis of costs and consequences of alternatives performed?


Were the additional (incremental) costs generated by the use of one alternative over another compared with the additional effects, benefits or utilities generated?

9.

Was a sensitivity analysis performed?

(a)

Was justification provided for the ranges of values (for key parameters) used in the sensitivity analysis?

(b)

Were the study results sensitive to changes in the values (within the assumed range)?

10.

Did the presentation and discussion of the results of the study include all issues of concern to users?

(a)

Were the conclusions of the analysis based on some overall index or ratio of costs to consequences (e.g. cost-effectiveness ratio)? If so, was the index interpreted intelligently or in a mechanistic fashion?

(b)

Were the results compared with those of other studies that had investigated the same questions?

(c)

Did the study discuss the generalizability of the results to other settings and patient/clinic groups?

(d)

Did the study allude to, or take account of, other important factors in the choice or decision under consideration (e.g. distribution of costs and consequences or relevant ethical issues)?

(e)

Did the study discuss issues of implementation, such as the feasibility of adopting the 'preferred' programme, given existing financial or other constraints, and whether any freed resources could be used for other worthwhile programmes?

Source: Drummond MF and Stoddart GL (1985) Principles of Economic Evaluation of Health Programmes World Health Statistics Quarterly 38(4): p 365.