Cover Image
close this bookParticipatory Impact Monitoring - PIM Booklet 2: NGO-Based Impact Monitoring (GTZ, 1996, 38 p.)
close this folder2. NGO-based impact monitoring
View the document2.1 Advantages and obstacles
View the document2.2 Steps in introducing and carrying out NGO-based monitoring of socio-cultural impacts
View the document2.3 Joint reflection workshops
View the document2.4 Facilitating the PIM process

2.1 Advantages and obstacles

At the beginning of this booklet, we listed some conditions which should be fulfilled for PIM to be applied successfully. We then described the basic ideas underlying PIM. As we have seen above, there are advantages in theory, but perhaps obstacles in practice.

PIM holds out the promise of being useful - but certain conditions must be fulfilled:

PIM is an appropriate tool for managing a self-help project but do you really want to try a new monitoring concept? PIM may take more time but do your staff members want to spend more time on joint decisions?

PIM is intended to empower people who have no voice but will the people in power at the moment accept a loss of influence?

PIM makes the project and the organization more transparent - but is this transparency really desired?

PIM may cause some conflicts if there are divergencies between attitudes, expectations and objectives. Are you ready to confront them?

Is your organization strong enough to tackle this new task? Are the leaders and members willing to face the hurdles mentioned above? If so, it is very likely that you will be rewarded with fruitful improvements of your activities!

There are several other good reasons why an NGO should adopt PIM:

- In an NGO there is a lot of knowledge concerning socio-cultural impact lying fallow. The staff have a large body of experience with learning processes in similar groups or projects. Field workers continuously observe the changes in their clients' environment. In order to improve project management, NGO-based impact monitoring could mobilize and systematize this knowledge.

- In some organizations, the results of the field staff's and social workers' endeavours are not perceived or appreciated, especially when they are "invisible" rather than technical or economic. NGO-based impact monitoring could contribute to personnel guidance and team development by demonstrating the value of field work.

- Non-profit organizations do not have economic, but intangible objectives. Their performance is measured not by economic indicators (like profit or turnover) but primarily by social, cultural or other qualitative criteria. NGO-based impact monitoring can also be important in managing the development of an entire organization.

If there are conflicts, open or hidden, PIM helps to bring them to the surface. Misunderstandings can then be resolved if the actors are ready for open dialogue.

While NGO-based PIM is equivalent to group-based PIM, it is not merely a copy of it with different actors. The purpose of NGO-based PIM is to accompany group-based PIM and, to some extent, to complement it.

NGO-based impact monitoring consists of three elements:

A. Monitoring of socio-cultural impacts is similar to group-based PIM (see booklet 1). The differences are:

- the field workers are the main actors, they select and specify indicators, observe, document and analyze the changes, and make (or prepare) decisions;

- monitoring focusses even more on socio-cultural impacts, i.e. learning processes, capacity building, changes in behaviour.

B. Joint Reflection Workshops are regular joint meetings of the NGO and the group. Results of group-based impact monitoring are compared with NGO-based monitoring of socio-cultural impacts.

C. Facilitation of the PIM process and ongoing accompaniment of the group by the development organization or NGO are necessary for the introduction and functioning of PIM.

Introducing PIM to NGO field staff

Before introducing PIM in a self-help group, it is crucial that an in-depth introduction to the PIM concept be given to the NGO staff involved, as they will be part and parcel of the implementation. This introduction to PIM, however, should be initiated carefully, starting with the field workers' practical experience.

Therefore, the introduction to PIM should not start with a theoretical explanation of monitoring or impact. The field staff should be encouraged to bring in their own experience and ideas. It must be made clear that their experience is valued highly - so start with questions:

- What important changes for the people has your work induced?
- Which changes are normally reported on? Which changes are often ignored?
- What has changed in people's behaviour? What have they learned?
- Have other groups learned from these experiences?
- Is it possible to find simple indicators for these changes?
- How far were these indicators observed by the group members?

PIM is introduced through workshops. Three sessions will probably be necessary to sensitize the staff and explain the concept. Methodologically, the introduction could contain the following:

1) field staff's observations, ideas and experience on monitoring, impact, participation
2) short introductory speech
3) short handouts using visualization, pictures, examples
4) detailed discussions on advantages and risks of PIM to field staff and group
5) adaptation of the proposed concept to the actual work of the field staff

Only after the field workers' concrete examples of socio-cultural impact have been collected and analyzed should a more theoretical introduction to PIM be given. It will be very important to listen to and respect the field staff's comments concerning

- the viability of the concept,
- the additional workload, and
- the usefulness of impact monitoring.

PIM does not ask for scientific definitions, but for subjectively important changes. The various actors have different views which need not be contradictory, but which are interlinked and have to be compared.

2.2 Steps in introducing and carrying out NGO-based monitoring of socio-cultural impacts

Preliminary Step: What do we know about the context?
Step by Step Step l: What should be watched?
Step 2: How can it be watched?
Step 3: Who should watch?
Step 4: How can the results be documented?

Which information and for whom? When and how?

Steps 5 to 7: What did we observe?
Why? What should be done?

As mentioned above, the procedure for NGO-based monitoring of socio-cultural impacts is similar to that of group-based impact monitoring, which was described in Booklet 1. The following description is therefore merely a brief outline.

Preliminary Step: What do we know about the context?

Certain essential information about the situational context should be available before PIM is introduced. It is then easier to adapt PIM to specific needs and integrate it in a given context.

Apart from this, if possible, you should as a rule try to use participatory methods for situation analysis which also serve for planning, monitoring and evaluation, such as

- PRA: Participatory Rapid Appraisal
- PAR: Participatory Action Research
- SWOT: Strengths, Weaknesses, Opportunities, Threats
- PALM: Participatory Learning Methods
- GRAAP: See, Reflect, Act (with the help of pictures)

These methods are based on ideas similar to PIM. You should make use of these to permit a realistic assessment of how people see their situation, their problems and needs.

STEP BY STEP

Steps in introducing PIM

1. What should be watched?

expectations and fears of the staff members with regard to socio-cultural changes

2. How can it be watched?

concrete examples of how these changes


can be observed (indicators)

3. Who should watch?

elected staff members who are directly


involved in the respective project

4. How can the results

records, tables, graphs, descriptions charts, be documented?

Steps in carrying out PIM


5. What did we observe?

reports at the beginning of staff meetings

6. Why do we have these

assessment and analysis by the staff results?

7. What should we do?

immediate decision (or preparation for a decision) at the meeting (= adjustment of plan)

Step 1: What should be watched?


Expectations and fears

The project team like the self-help group should make a note of some of the expectations and fears concerning the self-help project. As far as possible, these should relate to the socio-cultural impacts: skills and learning processes in the group. In this context the following questions are helpful:

- Based on your experience, what socio-cultural changes do you expect or fear from this project?

- What socio-cultural impacts resulted from similar projects?

- Which impacts were barely registered by conventional monitoring instruments?

- On the basis of which factors were they clearly recognizable?

- Which socio-cultural indicators should we be aware of in order to manage this project?

In this way, some of the hypotheses on the future development will be identified. It makes sense to discuss these hypotheses and indicators with selected resource persons from the group.

This procedure may lead to a result which is identical to the objectives of the NGO, or to the overall goal, project goal and expected results of the project. If they are fully congruent, so much the better. Whereas in conventional monitoring we rely on the formal information of the planning documents, PIM relies more on an informal assessment by the field workers and NGO staff.

In the case of the store of the Housewives' Committees in Caracoles, Bolivia, staff members had a number of expectations and fears. The prioritized aspects included the following:

EXPECTATIONS

that prices would be lowered and better quality offered than in other shops
that the Housewives' Committees would take over responsibilities
that a higher level of integration of the three Caracoles cooperatives would be achieved
that an opportunity would be created for women to participate in the cooperatives

FEARS /DOUBTS

Should they sell for cash only or give credit?
that they would not be able to recover the credits
that the cooperatives might not pay the store

that the women might not be able to administer the store themselves

Step 2: How can it be watched?

"Monitoring" and "indicators" are often quite abstract terms. Those responsible for keeping a watch should be encouraged to report very simply, on the basis of their experience, how they can tell that people have learned and changed. The procedure for mobilizing the practitioners experience should be similar to group-based impact monitoring. Concrete examples of how the social environment changes can be presented by each team member.

The reasons for these changes can then be analyzed and the most vivid examples chosen for illustrative purposes. In accordance with what was said concerning group-based monitoring, there is no fixed procedure for deriving indicators from these examples. It is not a problem if no measurable or scalable indicators can be found, because in NGO-based impact monitoring descriptive examples can be observed and documented as well.

The expectations and fears to be observed may be chosen in Step 1, or definitively decided upon here in Step 2.


Figure

In view of the work burdens project teams are faced with, it is not necessary initially to introduce more than three to five indicators for a project. The necessary number of indicators, however, will depend on the complexity of the organization and its activities and hence its monitoring system.

PIM is easy to link to a conventional monitoring system. If the NGO has one, it must determine how far the socio-cultural indicators should be integrated into it.

In Caracoles, the NGO staff selected the following indicators for observation:

EXPECTATIONS/FEARS

that prices would be lowered and better quality offered
than in other shops that they would not be able to recover the credits
Should they sell for cash only or give credit?
that the cooperatives might not pay the store
that a higher level of integration of the three Caracoles cooperatives would be achieved
that the Housewives' Committees would take over responsibilities

INDICATORS

the prices of 20 staples in the shop are below prices in other shops nearby
credits given each month do not exceed the cash payments received
the cooperative leaders held monthly meetings to analyze results
the committees were able to check the discount efficiently

INDICATORS

(derived from expectations or fears)
the prices of 20 staples in the shop are below prices in other shops nearby
credits given each month do not exceed the cash payments received
the cooperative leaders held monthly meetings to analyze results
the committees were able to check the discount efficiently

OBSERVATION METHODS

list the prices of each staple monthly for the shop and for 5 other shops nearby separately; reply YES or NO and note down what was observed and what comments were made.
list credits granted and payments received; reply YES or NO and note down what was observed and what comments were made

reply YES or NO and note down what was observed and what comments were made

price lists and prices charged are checked monthly by the committees; reply YES or NO and note down what was observed and what comments were made

Step 3: Who should watch?

The socio-cultural impacts chosen as indicators are frequently those which have already been observed by the NGO personnel. It is therefore best for the field workers and other NGO staff members to observe the selected changes themselves. One or two people should be chosen to be responsible for observation.

However, it is also useful to confirm one's own views by cooperating with other people or organizations who know the project environment: a teacher, a priest, or staff members of other NGOs, or any insider concerning the group's internal structures. Whether this is appropriate will depend very much on the specific conditions.

While it is not the purpose of NGO-based PIM to employ members of self-help groups as observers, they need not be excluded. It is first and foremost the field staffs' view which is of interest here. The views of the self-help group and NGO will be compared later in the joint reflection workshops this is part of the learning process!


Figure

Step 4: How can the results be documented?
As explained for group-based impact monitoring, a record must be kept of the impact observed. If it is done in the same way as for conventional monitoring, it will be recorded in a kind of logbook.

In the Caracoles project, monitoring forms were developed for noting down the prices observed in the various shops each month. For other indicators, there are questions which have to be answered with yes or no, with a blank space for remarks and comments.

Graphs and charts are also helpful for visualizing quantitative indicators. Indicator no. 2, about credits granted and payments received, was drawn as a bar chart.


Figure

Which information and for whom? When and how?

The NGO has to set priorities in information flow in accordance with the decision-making structures. This means that not all information has to flow to the NGO director or even to the funding agency, only summaries from time to time, or to report outstanding successes or conspicuous failures.

These rules for information flow should be worked out jointly by the entire NGO team. They should also decide what kind of monitoring information is regarded as sensitive or confidential, and establish clear rules as to how it should be handled: who must be excluded from the information flow?

In addition, a decision must be taken on how information should be fed back to the group. In Caracoles, the women were informed monthly. A minimum would be reports at the joint reflection workshop.


Figure

Important: All those involved must accept that although each organization wants to know what the others are doing behind the scenes, every actor is entitled to confidential treatment of his inside information. Not everything has to be analyzed: some secrets are best left under wraps!

Steps 5 to 7: What did we observe? Why? What should we do?

The steps to answer these questions are now basically similar to the steps described in Booklet I (Group-based Impact Monitoring).

Also, an NGO might already have a (conventional) monitoring system. Socio-cultural impacts might be monitored in a similar way. PIM is compatible with general management rules.

As mentioned in Section 1.2, the depths and periodicities of reflection and decision-making are different in every organization: you should match your PIM rules to your project and organization structure.


Figure

2.3 Joint reflection workshops

Formulation of guiding questions

Workshop Step 1: What has changed?
Workshop Step 2: What have people learned?
Workshop Step 3: What action must be taken?
Workshop Step 4: How can we improve our impact monitoring?
Post-Workshop Step (5): What conclusions can we draw for our work?
Concluding remarks

PIM comprises different autonomous monitoring schemes of various actors (self-help group, NGO, FA or other organizations/groups) in a single project. The individual actors observe the area that interests them most.

Advantages:

- it is not necessary for everyone to gather all the data, and the amount of data stays manageable for each actor;

- the facts are seen from different perspectives;

- more information is available for joint decision-making.

These different perspectives complement each other, and can portray the project reality more completely and realistically than a single actor could. To this end, the observations of those involved must be communicated and discussed regularly. The Joint Reflection Workshops fulfil this important need.

They provide a forum for exchanging and evaluating information. The various actors also hold a mirror up to each other, enabling them to compare the way they see themselves with the way others see them.

The Joint Reflection Workshops are held regularly together with the self-help groups. They will be organized by the NGO to reflect on the progress of the project, though not so much in the sense of the planned "project outputs" as in the sense of impact assessment. These workshops should be arranged in a manner and an environment that the self-help group is familiar with. The participants are the parties involved (for ex.):

- self-help group
- NGO field staff
- NGO office staff


Planned project

Procedure at the joint reflection workshop (4 guiding questions):

Step 1: Compare observations: "What has changed?" Step 2: Analyze socio-cultural impacts: "What have people learned?" Step 3: Take decisions: "What action must be taken?" Step 4: Improve monitoring (if necessary): "How can we improve our impact monitoring?"

After the workshop:

Step 5: Evaluate internally: "What conclusions can we draw for our work?" If necessary take decisions on changes.

The frequency will depend on the degree of familiarity between the NGO and the group, and on the importance of the project. At first it might be every three months, in the long run at least once a year.

Reflection should not be limited to these rare workshops; it is also customary, of course, in routine project work and at meetings. But as described in the introduction, it is useful to have different opportunities for reflection, with different depths and frequencies.

Normally, the day-to-day problems of a project tend to dominate meetings between field workers and the self-help group. However, at these monitoring workshops the participants should explicitly try to look back to the start of their activities, comparing it with where they stand now. In a process involving prolonged activity this is an opportunity to stop for a moment for profound reflection.

It is advisable to have an independent facilitator who has the confidence of all sides.
Formulation of guiding questions

The field staff and the group will have to discuss four guiding questions. These questions must be discussed with the group by adequate methods (i.e. not simple questions and answers).

The results obtained by asking these guiding questions will lead to an analysis of the project context. Emphasis should be given to socio-cultural impacts. The discussion may induce decisions concerning the project's activities, or at least pave the way for decisions which have to be taken by superiors. If necessary, the monitoring system will be revised.

The questions must be open; the NGO should not influence the replies. Although NGO staff and field workers might have their answers to the questions, it is still important that they should first ask the group, by appropriate methods, and only afterwards encourage discussion by introducing their own observations (if necessary).

Workshop Step 1: What has changed?

The monitoring workshops start with (Step 1) the general guiding question "What has charged?".

This is to compare the results of the group-based monitoring system with the the results of the NGO-based monitoring system.


Comparison of observations

The general question "What has charged?" leads to some deeper questions. Some questions refer to change:

What/Who has changed?

(This question is meant to introduce the report on the group's findings, which are a result of the group-based monitoring system.)

What has caused the change?
(the individual members, the NGO, or other factors?)
How has it changed?
How has this change affected you?
What other change(s) has/have occurred as a result?

Clearly, these questions cannot be limited to socio-cultural impact; they include everything that is important to the group. The discussion should nevertheless focus on socio-cultural impact.

Workshop Step 2: What have people learned?

When analyzing the socio-cultural impact, the NGO personnel not only have to refer to the indicators formulated in their monitoring system. They should try to grasp the various learning processes as a whole, and this is perhaps more feasible by asking open questions than by a strict comparison of isolated indicators. NGO members should be aware of the discussion needs of the group and use the opportunity of the joint reflection workshop for an open dialogue with the group.

What have people learned?

- Have the members of the group taken on new responsibilities?
- How far have the group's internal and external relationships changed?
- How far has the internal structure of the group changed?
- What new activities have been started by the group (or by members of the group)?
- What similar activities have other groups (or individuals) started?


Figure

Workshop Step 3: What action must be taken?

The next step is decision-making. The analysis of the findings will be aimed at achieving unequivocal results here. In keeping with the importance and frequency of the Joint Reflection Workshop, the decisions taken here tend to be of a strategic nature. That is to say, they indicate the basic direction and provide a framework for the solution. Operational decisions should be taken subsequently, at other meetings.

What action must be taken?

- What should the members of the group do?
- What should the project team do?
- What should the other people involved do?
- Who else should be brought in?

Workshop Step 4: How tan we improve our impact monitoring?

The last step in the workshop is fairly general: if important issues have previously been neglected the monitoring system must be revised. In such cases it is useful for all those involved to agree at the workshop that these issues should be included in the monitoring system. Alternatively, each organization may take a decision at its own evaluation meeting after the workshop.

Potential improvements in PIM

- Which criteria and indicators should be improved?
- Which criteria and indicators are no longer necessary?
- How could the observation and assessment system be improved?
- How did you feel in our reflection workshop?

The case studies from the PIM field phase showed that many expectations/fears and their indicators which had been identified in the first meetings tended to be of short-term interest. By revising the monitoring system periodically, aspects which are of long-term interest are automatically sifted - and thus relevant indicators for the sustainability of the project come to the fore.

Post-Workshop Step 5: What conclusions can we draw for our work?


Figure

After the monitoring workshop, the NGO should internally evaluate the results of the NGO-based impact monitoring. This reflection should go beyond the management of the actual project: instead, it should refer to more fundamental questions which often relate to the development of your own organization. If changes are necessary, decisions should be taken immediately.

The following questions should be discussed in the NGO after the workshop:

With regard to project management:

- What have we achieved? How have we achieved it? Who has assisted us? What has helped us to achieve this?

- What conclusions can we draw from the comparison between the group's observations and our own?

- Does anything need to be changed in our activities?

With regard to PIM:

- Which socio-cultural indicators need to be taken into account in our own regular monitoring and evaluation?

- How well did the original proposal work?
Has it been modified by the beneficiaries' opinions?

- Should a questionnaire be circulated at regular intervals to document the specific activities of the projects, beneficiary groups and the context?

With regard to your own NGO:

What can we learn from this project that should also be considered in other projects?

- Should we introduce new internal rules to improve our cooperation?

Concluding remarks

For the field staff, it will be important to learn about people's perception of their work. The participation of an external facilitator will further reinforce this. The feedback is useful for the field worker's self-assessment. Joint analysis of observed changes by the people's group and the field staff will increase the appreciation of successes which were previously concealed or rated as "merely subjective" impressions.

2.4 Facilitating the PIM process

Generally, PIM will be introduced in a group or people's organization on the initiative of the NGO. To a certain extent, therefore, the promoters are responsible for ensuring that group-based impact monitoring works.

For the NGO staff, it should again be emphasized that all of the group-based PIM and most of the impact analysis at the Joint Reflection Workshop is done by the group. The facilitator acts as catalyst. S/he does not tell people how to interpret the results of observation, but if necessary s/he will guide the group by asking questions. For group-based impact monitoring, it is crucially important that the community (not the NGO's field staff ) identifies with the analysis.

You should not risk nipping group-based impact monitoring in the bud due to disagreements. If the group's and the NGO's observations differ, other opportunities should be provided or created for analyzing the content and the causes (e.g., the post-workshop step). If the staff member cannot cope with the conflict of roles, i.e. acting both as facilitator and as NGO representative, other facilitators must be found.

How can PIM be introduced into the self-help group?

Especially at the beginning, it is uncertain whether the idea has been well explained and understood. How do you carry on after the expectations and fears have been identified? To find out what is important, to find observable indicators, to find good observers, to observe, to feed back the results to group meetings - all this is easily said but not so easily done.

The NGO staff involved in PIM will probably have to stay in the village or with the group for several days when introducing the basic steps for the first time. During the first few weeks after setting up the first "draft" of group-based impact monitoring, the observers will need support. Presentation of the findings at group meetings should be facilitated by NGO personnel for the first year at least. How this is done is shown in examples from the field phase given in Booklet 3.

This may seem to be a lot of additional work; but if PIM is introduced when a new project is implemented, when we assume that frequent visits wild be needed in any case, extra visits specifically for PIM will probably be unnecessary. And don't forget: PIM will definitely also help save time because


Action without reflection is a waste of time!

We need the cooperation of other practitioners and thinkers to test and to improve PIM.
If you are implementing and testing PIM in your project area we would be very interested to hear from you. Write and tell us about your experience with PIM.

We are planning to organize more regular and more efficient exchanges, if a substantial number of practitioners continue with the development and adaptation of PIM.

Please write to

FAKT
Association for Appropriate Technologies
Gheidestral 43
D - 70184 Stuttgart, Germany

or

GTZ- GATE (ISAT)
German Appropriate Technology Exchange
Postfach 5180
D - 65726 Eschborn, Germany

Thank You