Cover Image
close this book Boiling Point No. 23 - December 1990
View the document Measures of success
View the document Methods of Monitoring & Evaluation of Stove Programmes
View the document Measuring the Successes and Setbacks
View the document Improved Stoves, Women & Domestic Energy
View the document Monitoring & Evaluation?
View the document Bringing Stoves to the People
View the document Why use Technology Assessment when Implementing Technological Change?
View the document Two Stove Programme Alternatives
View the document Product Quality Monitoring
View the document UNEP/Bellerive Kenya Stove Programme
View the document Stove Programmes in Sri Lanka: Reflections on the First Decade
View the document Gate/GZT news
View the document Solving Sampling Problems in Khartoum
View the document Technology at Ky Anh
View the document Building and Using an Efficient Cookstove
View the document Enclosed Traditional Brushwood Kiln
Open this folder and view contents News
Open this folder and view contents Publication
View the document Letters to the editor
View the document Acknowledgments

Measuring the Successes and Setbacks

by Emma Crewe (Social Anthropologist, ITDG)

Deaths caused by respiratory diseases. If fuel consumption and smoke were reduced by 30% in the laboratory, and a few educated guesses could be made about benefits, then no more questions were needed. Apparently, if the potential users seemed a little sluggish in snapping UP the new stove. they merely needed to be educated to realise what they wanted. Where targets had to be met, and lower income groups were supposed to buy the stove, subsidies were often introduced. The result has been high dissemination rates, but little evidence that the stoves survive. Stove programmes need more realistic objectives and easy ways to measure successes and set backs.

Since it has become clear that a techno-fix approach to development is not going to eliminate poverty in the South, the development business has attracted social scientists. In stoves work the conventional approach is to identify an improved stove and conduct a collection of appraisal studies

Why bother with monitoring?

It has been claimed that smokeless or fuel-efficient stoves can alleviate poverty, save trees, or prevent (stove performance tests, household field tests, andReviewing monitoring and evaluation maybe some market research), carry out a stove acceptability survey and compare the number of stovesIn preparation for the workshop, a review of stoves disseminated to targets. Even though it has beenM&E was initiated. For example, Simon Burne accepted that wood-burning stoves for rural dweller spointed out at the 1987 FWD/CEMAT Guatemala do not save trees, and reduced fuel consumption is notConference that mainstream economics relies on the reading priority for all users, estimating fuel savingsunrealistic models and is hopelessly ill-equipped to has dominated monitoring. Is there a more imaginative way of monitoring projects so that technology responds to people rather than the other way around? Improving monitoring and evaluation GTZ, ITDG and FWD have embarked on a project to improve monitoring and evaluation (M&E) so that project staff increase their efficiency in working with users and producers and to deal with social forces outside the market economy. Payback periods, discounted cash flow and break-even analysis can be convenient for economists because development programmes can be reduced to simple monetary facts and figures. However, once it is recognised that programmes are about processes and relationships, the people rather than money can be placed centre stage. Burne has designed

a. "a decisions matrix" obtain evidence of Participant at the Arnsha Workshop envisaged that three phases would tee necessary:which enables decision-makers to appraise thedifferent production and distribution options while :

1. Planning - agree on a common M&E system and plan bearing in mind the conflicting interests of both users the field to stand producers (see 'Stoves For People', IT)

2. Field-testing - try out the planned system to ensure Publications, London, 1987, pp.129- 133) it is workable and effective.

3. Review - review findings, revise the system for FAO has also made an extremely valuable publication,and consider "raining strategies"contribution to M&E guidance. In 1985 FAO produced the draft 'Guidelines for the Monitoring

To carry out phase 1., a workshop was organized in Pilot Stove Development Schemes'. Under the Arusha, Tanzania to bring together the co-ordinators auspices of FOOD, seven stove programmes carried out from GTZ/ITDG/FWD, project staff who would be evaluation studies, one objective of which was to invited to test the system,and representatives from the refine the methodologies laid down in the FAO East-West Center (Hawaii) and Association publication. (See 'Bringing stoves to the People: An Bois-De-Feu (France) who are also engaged in stoves Assessment of Impact, ACTS, Nairobi, 1990, reviewed M&E work. Participants from India, Pakistan, Sri in this edition of BP). FAO will publish a set of Lanka, Tanzania, Kenya, Zimbabwe, Guatemala, guidelines on a wide range of M&E areas, including USA, France, Germany and UK attended, of whom 7 definition, planning data collection, collecting and were women and 7 were men. The workshop, which storing data, analysis, training and feedback. Samples took place between 23-27 July 1990, in Arusha of questionnaires, procedures for tests, and guidelines Tanzania was organised by CARMATEC/GTZ and on participatory methods will be included. ITDG, Rugby and funded by GTZ (Household Energy Section), Frankfurt.Before the Arusha workshop each participant wrote a paper on the main strengths and weaknesses of current methodologies.

Some of the main points were:

• While carrying out a survey in Colombo, Sri Lanka, users were too polite to criticise the stoves which had been given by the protect;

• The constant follow-ups in the Women and Energy Project in Kenya have been particularly instructive since users' complaints and suggestions led to alterations in stove design, and even models, (such as the withdrawal of the two-pot stoves and changes in the position of the pot-rests).

• Recommendations have emerged from ASTRA's M&E of the stoves programme in Karnataka, but the level of implementation of these has been marginal (see Dr. Ravindranth's article in this edition of Boiling Point).

• Insufficient attention has been paid to small-scale enterprise developments, as examples: the number of distributors and sales outlets; the differences between small- and large-scale producers; and the profitability of stove production.

• There is a need for: training in monitoring; clarity on which areas should be monitored internally and which externally, M&E to be planned but able to adapt to changing project plans; realistic recommendations; and M&E to be seen as constructive self-riticism rather than a threatening secret service mission.

Many of the M&E needs identified by project staff during the workshop will be fulfilled by the FAO guidelines. There is one particular element which the FAO Guide will not address:- a range of indicators for common objectives. The FAO guides tells us how to monitor; we also need to decide what to monitor.

A new framework

Workshop participants began devising a framework for indicators which relate to:

1. different objectives;

2. disciplines or subject areas;

3. process/impact/context (these relate to the terminology used in Goal Orientated Project Planning which stipulates that 'process' includes what happens as a result of project activities, 'impact' involves the effects of these results, and the 'context' is made up of factors which affect the project, but which can not be controlled).

The result was that stove programme objectives were divided into five areas: managerial, technical, economic, social, and environmental. Within each area, all the possible realistic objectives were listed. Under each objective, indicators will be presented in order of priority, with the methods for collecting information. In effect, the framework will work like a menu - so that project staff can turn to those objectives which they are concerned with.

To give an example within the area of environment, objectives might be found under the following headings: - Pressure on Biomass Resources, Health, Safety, Greenhouse Gases, Fuel Use, Environmental Policy and Legislation, and Awareness Creation. (It was decided that reducing the rate of deforestation was not feasible for a stove programme). Under each objective heading, methods for tracking essential indicators will be given ea., if the project aims to reduce Pressure on Biomass Resources, these might be monitored: - household fuel consumption (per day/week) and level of agricultural residues and dung use as fertiliser (for impact); and fuel prices and the rate of deforestation (for context). (Measuring household fuel consumption per unit of capital, and household fuel consumption as a control group, would only be possible for a research project). The complete manual will have indicators listed under 34 objectives. There will also be indices which point to methods relating to categories (process/impact/context) and another one to explain which parts will interest different manual users (donors/project staff/beneficiaries/ monitors).

Once the framework for the M&E menu was drafted, participants drew up criteria for selecting the projects which would be invited to try out the guidelines. Seven participants at the workshop agreed to co-ordinate M&E trials either through their own organisations or with another in their region. In addition, three projects will be given the manual, to see if it is comprehensible to those not involved in the process of devising it. All ten project representatives will be brought together in early 1992, at the end of the trials, to revise the guide according to any recommendations which emerge out of the testing.

Any other stove programme staff interested in conducting a test of the guide are more than welcome. Advice from a wide range of projects is urgently needed to ensure that the choice of indicators provides relevant elements for all stove programmes, irrespective of location and scale. (If this proves impossible - producing booklets on a regional basis may be considered). The guide will be ready for distribution by January 1991 and will be available from ITDG free on request.

(Proceedings of the workshop, papers by participants and a bibliography of useful publications on monitoring and evaluation is available from Ms T Flavell, Fuel For Food Office, ITDG).