Cover Image
close this bookNon-formal Education Training Module (Peace Corps, 1991, 182 p.)
View the document(introduction...)
View the documentAcknowledgements
View the documentPreface
View the documentIntroduction
View the documentWhat is Nonformal Education?
View the documentAdult Learning
View the documentSession: 3 Helping People Identify Their Needs
View the documentFacilitation Skills - Part 1
View the documentFacilitation Skills - Part 2
View the documentNFE Materials Development
View the documentGames in NFE
View the documentPlanning
View the documentEvaluation
View the documentLooking Back/Looking Ahead
View the documentAppendix I: Warm- Ups
View the documentEvaluations
View the documentReferences

Evaluation

Rationale

Evaluation for NFE involves participation of local people in planning, designing and carrying out the analysis of their own projects. To help them do this, Volunteers need to know how to sort out the complex questions involved in evaluation and to think about how to apply techniques they may be familiar with to specific evaluation situations. They also need experience designing and carrying out an evaluation themselves. In this session they will design evaluations for the NFE workshop or for their own NFE activities.

Objectives of Session

· To explore ways of involving local people in evaluating their own activities.
· To choose appropriate evaluation techniques for specific situations.
· To design an evaluation for an NFE activity.

Activity Sequence

1. Warm-up

10 minutes

2. Who Wants To Know What For What Purpose?

10 minutes

3. Evaluation Techniques

45 minutes

BREAK

15 minutes

4. Evaluation Design


Design

45 minutes

Critique

30 minutes

Modifications

15 minutes

5. Evaluation of Session

10 minutes

Total Tune Required

180 minutes

Materials Needed

· Flip chart paper
· Markers
· Hand-outs:

Case Study from Session 8, Waste Water Gardens - One per participant
Evaluation Problems - One per participant
Types of Evaluations - One per participant

Trainer Preparation

1. Read Peace Corps NFE Manual, Chapter 6.

2. Read through the session with your co-trainers and decide together on the options that you want to use.

3. Write new case study and evaluation problems if necessary. See Activity 3, OPTION, page 150.

4. Assemble all materials.

5. Be sure that the participants who signed up to do the warm-up and evaluation have the materials they need and are ready.

Activity 1: Warm-up

Activity Time 10 minutes

Purpose To explore different ideas about evaluation.

Step - by - Step

1. Ask the group:

What words, or phrases, or mental pictures come to your mind when you think of the word "evaluation?"

Write responses on flip chart paper (Examples: "a test," "a doctor's office,. "letting the facilitator know that the session was too long, etc.). Try to elicit examples of both evaluation by authorities and evaluation of the authorities.

2. Ask the group which of the words, phrases or descriptions they would say are "top down" evaluation (e.g., evaluation by those who hold more power of those who hold less power). (Examples: teachers grading students' performance, or project sponsors evaluating the level of health in a community after a vaccination campaign).

Ask the group which are examples of evaluation where decision-making power is balanced more equally. (Examples: participants and facilitator evaluating a session together, or a neighborhood association evaluating a project sponsored by an outside agency. )

NOTE

It may be useful to stress that "top-down" and "participatory evaluation are two ends of a continuum, and that participants' examples may fall at many points in between. Even within participatory evaluation there may be many levels of participation, ranging from the beneficiaries of a project being asked to give their opinions of it to greater levels of involvement such as deciding on evaluation criteria, carrying out the evaluation, interpreting the results, etc.

It may also be important to bring out that evaluation by project sponsors is not necessarily an abuse of power (when it is done respectfully and fairly), and evaluation by participants is not necessarily constructive (when it is done simply to vent feelings of frustration rather than to work together to change the situation).

3. Let the group know that in NFE, we are concerned mainly with evaluation in which there is a high level of involvement of the project beneficiaries. You might want to refer participants to the definition of NFE from Session 1 and ask them why they think this is so. (Note especially the last part of the definition: "NFE comes from the people, rather than being taken to them.")

Activity 2: Who Wants to Know What for What Purpose? (WWP)

Activity Time 10 minutes

Purpose To introduce the group to the central question to consider in designing any evaluation.

Step - by - Step

1. Let the group know that in this session, they will think of some evaluation strategies and then apply them in designing their own evaluation. But in order to design any evaluation, it is useful to ask the question: "Who wants to know what for what purpose?. Write this question on the board or on flip chart paper.

2. Ask participants to recall one of the daily session evaluations. Ask the group to answer the question for that evaluation. Repeat with another example with a different "what?"

(For example, one evaluation may have been designed to elicit participants' feelings about a session; another may have asked for specific suggestions for improvement; another may have asked participants to say what they were confused about, etc.)

3. Ask for other examples of evaluations, both "top-down" and with participation, in the participants' experience (language exam, self-evaluation of progress toward goals, etc.).

Ask the group to say who wants to know what for what purpose for each of the examples.

NOTE:

Participants may come up with some examples of evaluations with multiple purposes (For example, an evaluation of an NFE E project funded by a development agency may need to serve the pragmatic purpose of renewing the grant and at the same time, help the project planners discover how to improve the things that went wrong). This is an ideal time to stress the importance of asking the "WWP." question when designing an evaluation in order to avoid conflicting purposes. Suggest to the group that they may have to design two separate evaluations to serve the two purposes as best they can.

Activity 3: Evaluation Techniques

Activity Time 45 minutes

Purpose To apply evaluation techniques appropriately to NFE activities.

Step - by - Step

1. Let the group know that after they decide on the WWP, they will then need to choose evaluation techniques appropriate to the situation they are evaluating. Ask the group to brainstorm as many evaluation techniques as they can think of. (Examples: questionnaire, interview, etc.) Encourage them to break down larger categories (e.g., questionnaire) into specific types (e.g., open-ended questions, yes/no questions, scale of 1 to 5, etc.). Write the group's responses on flip chart paper and post. This activity should take 15 - 20 minutes.

NOTE: You may want to use the evaluation techniques on pp. 108-109 in the Peace Corps NFE Manual as a guide to see that a variety of techniques are brought out. You might suggest other techniques from this list if participants have missed them.

2. Let the group know that they will now use the techniques they have generated to apply to some specific examples where evaluation might be necessary. Refer the group to the case study Waste-water Gardens they used in Session 8. If they do not have the case studies with them, you might read the case aloud or have a participant do so.

3. Say that you will give participants a handout that tells a little more about what happened in the waste-water gardens project and present some situations within the project that call for different evaluation techniques. Pass out the handout Evaluation Problems, and give the group a few minutes to read them.

4. Divide participants into three groups. Each group should choose one of the situations.

They should then take about 15 minutes to think of at least five evaluation techniques they could use to get at the same information. Refer them to the list of techniques they generated, above. Suggest that they guess what might come up in an evaluation in order to design appropriate techniques. For example, in Problem 1, they might wonder if the photos of successful gardens have provoked old rivalries in the community. Or, perhaps the bulletin board is not centrally located. Or, there might be too much written material and too few photos. Encourage the group to use what they know about their own local cultural context to guess at reasons for the problems in the project.

NOTE:

You may want to stress to PST participants how necessary it might be in your cultural context to use indirect techniques to get people to answer candidly. You might do this by asking the group, for

Problem 1, what answers they might get if they simply asked people, "Why aren't you looking at the bulletin board?" ("People might tell you what they think you want to hear. "People might be embarrassed, etc.)

5. Ask small groups to briefly report to the large group some of the evaluation techniques they chose to get at the information. Sum up the activity by saying something like this:

"By now, you have looked at: Who evaluates? What to evaluate? Why evaluate?

When to evaluate? How to evaluate? All of these questions are important to address when designing an evaluation.

OPTION

Instead of (or in addition to) having groups report the techniques they chose, you might ask them to report on one or two issues that came up in their discussions (For example, how to find out the opinions of women, the ethics of designing an evaluation to get funding, etc.).

You might want to use the results of this discussion to find out if participants feel they need to spend more time (e.g., an evening session) on evaluation techniques and problems. If so, ask for two or three volunteers to help you design an extra session. As part of this session you might invite a Volunteer and/or HCN to talk about their experience with evaluation and to share their techniques with the group.

OPTION

If you are doing Session 9 earlier in the workshop (e.g., before Session 8), you may want to use a different example of an NFE project, preferably one from your own cultural context. Write the case study, using Waste-Water Gardens as a guide as to length, amount and kinds of information, etc. Then write three or four evaluation problems that might arise at different points in the project. Use the handout Evaluation Problems as a guide. In writing case studies it is useful to try to answer the questions for participants yourself to see if your examples are clear, thoughtprovoking, and relevant to your cultural context.

BREAK 15 minutes

Activity 4: Evaluation Design

Activity Time 90 minutes

Purpose To design evaluations of the NFE workshop.

Step - by - Step

1. Let the group know that in this activity they will design two evaluations for the NFE Workshop. One will be a summative evaluation that they will administer in Session 10. The other will be an impact evaluation that can be administered some time in the future.

Ask the group if they know the meaning of the terms: formative, summative and impact evaluations. Have someone explain them to the group. You may give them the handout

Types of Evaluations to keep for future reference.

2. Ask participants to divide into two groups, one to design a summative evaluation that can be administered in 20 minutes and one to design the impact evaluation. Let them know they will have 45 minutes to design the evaluations. Then, each group will have 15 minutes to present their design to the other for a critique. Finally, both groups will take 10 minutes to make modifications to their evaluation designs.

3. Suggest that both groups start their evaluation design by asking "Who wants to know what for what purpose, and to decide on a "who," a "what." and a purpose for their evaluation.

They might then refer to the evaluation techniques they generated in Activity 3 to get some ideas for how to design their evaluations. They should also consider the question of who will administer the evaluation and how the results will be interpreted and

Session 9: I Page 151 reported to the trainer. If, for example, they need to print and photocopy a sheet of questions they should take responsibility for this, if possible.

NOTE:

You might convey to participants some things you as a trainer would really like to know and ask them to try to include these elements in their evaluation design.

NOTE:

In the next session there are 50 minutes set aside for the summative evaluation: 20 minutes to administer the evaluation, and 30 minutes to interpret the results as a large group and report them -in some fashion-to the trainer. If the group wants to use techniques that do not fit into this time frame (interviews, for example), try to give them the opportunity to do so.

OPTION

If the groups are large, they may decide to divide the tasks so that everyone has something to do. Or, they may decide to split their group in half and each design a separate evaluation. As part of the critique, they may try to incorporate the best features of each evaluation in a final version. They may decide to work on this after the session.

4. Keep time (45 minutes).

5. When the time is up, have each group present their design to the other. After (or during) each presentation, one of the group members should facilitate a discussion with the other group, asking for their suggestions and getting their final agreement on the design of the evaluation. Each presentation and group discussion should take about 15 minutes.

6. Let the groups know they have 10 minutes to redesign their evaluations if necessary.

Remind them that the group that designed the summative evaluation for the Workshop will administer it during the next session. Offer the help they may need to type out a final copy and duplicate it, if necessary.

NOTE: If there is substantial disagreement on either of the evaluation designs, participants may need to work after the session to negotiate a final version that everyone can agree on.

FOR IST:

Instead of (or in addition to) designing evaluations for the NFE Workshop, IST participants can work in small groups to design an evaluation for a project of one of the group members.

Or, all groups can work on one evaluation problem presented to the large group by one of the members. As a large group, participants decide on the WWP, then each small group works separately on an evaluation design. The participant who presented the problem is used as a resource person to give additional information to the small groups as needed. After a work period, all groups present their evaluation designs. The discussion that follows can focus on the differences and similarities of the evaluation designs, their feasibility, the possible uses of the information and how the design can be made more participatory.

Or, participants can work individually to design evaluations for their specific projects. After the work period, participants can pair up to share their designs and ask for suggestions and other feedback.

NOTE:

If you want to reenforce the Experiential Learning Cycle at this point, you might ask the group to recall the parts of this activity (design, critique, redesign) to see how they follow the "what?" "so what?, and "now what?, steps of the ELC.

Activity 5: Evaluation of Session

Activity Time 10 minutes

Purpose To have participants plan and carry out an evaluation of the session.

Step - by - Step

Have participants who signed up to evaluate the session to carry it out.

For Next Time

Ask for volunteers to form a committee to plan a simple reception for the HCNs who acted as resource people for the end of the workshop (see Session 10).

· End of Session 9.

Time Saver #1

1. Who wants to know What for What Purpose?

10 minutes

2. Evaluation Techniques

45 minutes

3. Evaluation Design

90 minutes

Total Time

145 minutes

Time Saver #2

1. Warm up

10 minutes

2. Who wants to know What for What Purpose?

10 minutes

3. Evaluation Design

90 minutes (Participants choose techniques from NFE Manual, pp. 108-109)

Total Time

110 minutes

Time Saver.#3

1. Warm-up

10 minutes

2. Who wants to know What for What Purpose?

10 minutes

3. Evaluation Techniques

60 minutes


(Take extra time for discussion of small group work. Trainer designs and administers evaluation of NFE Workshop)

4. Evaluation of Session

10 minutes

Total Time

90 minutes

Evaluation Problems for Case Study:

Waste-Water Gardens

1. John's group has made a bulletin board with photos of flourishing gardens and their proud owners. However, it is noticed that local people are not looking at the board as much as the group had expected. The group wonders why and what they can do about it. How could they find out what (if anything) is going wrong?

2. The three month garden project is over, and the group has noticed that only a few new gardens have appeared in their community. Though they are discouraged, someone points out that more people are talking about starting a garden next season. The group wonders, "Even though we didn't succeed in getting many people to plant gardens, is there something else positive that we have accomplished? Or were there problems that we didn't think of when designing our project? How can we find out?.

3. It is now a year later, at the end of the next planting season. The group has discovered that a problem with the project was the lack of garden tools. Many people who had learned how to use waste water simply did not have the tools necessary to hoe the dry, hard-packed earth. The group is now thinking of writing a small project proposal to solicit funds to make garden tools available at very low cost to the community. But in order to get these funds they need to demonstrate to the agency that the project so far has had some significant impact on the community. The group feels sure that the project has changed people's attitudes and increased their knowledge, even though there are not many new gardens. What evaluation techniques can they use to demonstrate this to the funding agency?

Types of Evaluations

A FORMATIVE evaluation is carried out while the project is going on. Its purpose is to let the group change course, decide on new activities, correct misunderstandings and work more effectively with the community.

A SUMMATIVE evaluation is done at the end of a project. Its purpose is to discover what went well or badly and if goals and objectives were met. It may be used to decide whether or not to repeat the project, or it may be used in a formative sense, to discover how to improve it for next time.

An IMPACT evaluation is usually done six months or more after a project is over. Its purpose is to discover the long term effects of a project on the lives of the people in the community. It looks not only at goals and objectives, but at the subtle, underlying changes that a project may bring to a community, and the impact of these changes over time.