(introductory text...)
TOTAL TIME: 2 hours, 30 minutes
OVERVIEW
By this session, participants have nearly completed all of the
steps in planning a health education project. Still remaining is the important
task of developing ways to evaluate the project.
Session 22 begins with a game that helps participants understand
evaluation and how it is conducted. Later, the group forms working pairs (as in
Session 20) to design specific evaluation criteria for their health education
project plans.
OBJECTIVES
· To define evaluation. (Steps 2,3)
·
To explain how and when to evaluate a project. (Step 3)
· To develop
and critique plans for evaluating a health education project. (Steps 4,
5)
RESOURCES
Bridging the Gap
Part IV
Demystifying
Evaluation
Helping Health Workers Learn. Chapter 9, pp.
12-22
Handouts:
- 14A Questions for Evaluating Community Participation
(From Session 14)
- 20A Planning a Community Health Project (From Session
20)
- 20B Health Education Project Planning Worksheet (From Session 20)
-
22A Criteria for the Evaluation of Strategies
- 22B Evaluation
Worksheet
MATERIALS
Newsprint, markers, forms and handouts from previous sessions.
PROCEDURE
|
Trainer Note
This session builds an information from Sessions 16-21. If these
sessions have not been used, you will need to expand the time in this session
and review the eight steps of the health education process in relation to
evaluation.
If you chose to do the "Hollow Squares" activity during Session 20
you may be able to delete Steps 1 and 2 of this session.
|
Step 1 (15 min)
Demystifying Evaluation
Ask the participants to select a game or contest with which they
are familiar (i.e., baseball, soccer, basketball). Have them form three small
groups and have one group represent members of the opposing teams, the second
group the fans, and the third group the umpires/referees. Based on these three
perspectives, ask each group to develop its evaluation criteria for deciding
which team wins the game when no score is kept.
Tell them their criteria can range anywhere from how handsome the
players are to how many times a player has committed a foul.
Inform them that they have 10 minutes to develop their list of
evaluation criteria for presentation in the next step.
|
Trainer Note
The purpose of this activity is to introduce the concept of
evaluation in a non-threatening way. Its objective is to reinforce or raise the
point that to determine the success or failure of a project (i.e., winning or
losing a game), people need to have a clear understanding to base their
decisions on (i.e., something measurable or that provides specific details of
what qualities are found in a winning team).
The reason for having the participants develop evaluation criteria
from the perspective of players, fans and umpires/referees, is to introduce the
notion that evaluation should involve people with different roles or
perspectives including:
- Active participants (i.e., team members)
- The
community at large (i.e., fans)
- Theoretically impartial observers (i.e.,
umpires/referees).
(An alternative to basing this activity on a sports game is to
adapt it to evaluating who wins a beauty contest.)
|
Step 2 (20 min)
Processing The Game
Ask a representative from each group to read and list on newsprint
the evaluation criteria they used in deciding which team won the game. After the
groups have presented their criteria, discuss the following:
- What kinds of things do their criteria measure
(physical qualities, ways the game is played)?
- Are each group's criteria basically the same?
- Do the criteria measure what is going on in the game (number of
people getting to third base) as well as how this is happening (players are
getting on base because the pitcher is walking them)?
- Would their criteria have been easier to develop or more
realistic if: the group had worked together? it was determined and stated at the
beginning what quantitative and qualitative factors usually affect or influence
the outcome of a game?
|
Trainer Note
When discussing criteria, the participants should understand that
objectives provide the criteria for evaluating the success or failure of
a program and how you are doing. Objectives must contain statements that are
measurable and observable and, prior to beginning a program (or game), what and
when you will evaluate needs to be determined. Without having some quantitative
or qualitative measures, this game in particular, and health programs in
general, become nebulous areas to evaluate. One final point is the need to
involve the participants in the program, and the community at large when
designing and evaluating the program.
|
Step 3 (45 min)
Determining How and What to
Evaluate
Introduce this step by telling the participants that evaluation
can address more than one aspect of a project. The terms "process" and "outcome"
are used to emphasize the generic purposes of evaluative efforts. Process
evaluation is aimed at improving a project and answers the question "Is our
strategy working?". Outcome evaluation is aimed at providing information for a
summary judgement of the project and answers the question, "Did we succeed?".
Distribute Handout 22A (Criteria for Evaluating Strategies) and
tell the participants, after they have read it, that same of these criteria
should be used for evaluating the strategies and related activities they select
for their projects.
Ask the participants to spend the next 10 minutes in work pairs
reviewing their Health Education Project Planning Worksheets and to think about
the following questions:
- Are the objectives stated in terms of what will be
measured and when?
- What information on the Project Planning Worksheet will
help them determine criteria for evaluation?
- What is the relationship
between monitoring and evaluation?
After 10 minutes, begin discussing the foregoing questions. Ask
someone to state:
- the purpose of evaluation
- information they need
to collect to do the evaluation and how they will collect it
- who should be
involved
- when the evaluation should be done
Distribute Handout 22B (Evaluation Worksheet). Using the example
from Handout 20B (Health Education Project Planning Worksheet), walk the group
through the Evaluation Worksheet.
|
Trainer Note
Participants should understand that evaluation can serve a number
of useful purposes including:
- assessing the needs of the community prior to
designing program objectives
- assessing the health educator's
performance
- assessing what participants learned
- assessing the cost
effectiveness of the project or activity
- assessing community
participation
- determining the extent to which their objective was
accomplished (i.e., did they train 10% of the community members to correctly mix
and use ORS solution.)
Be sure participants understand the need to evaluate every part of
a project (i.e., the strategies and related activities they have planned as the
means to attain their objective) to be able to pinpoint strengths and weaknesses
and make appropriate modifications. The evaluation criteria they should use when
assessing strategies and related activities are adequacy, appropriateness,
effectiveness and efficiency. These terms are defined in Handout 22A. Make sure
the participants are comfortable with these terms.
By the end of this discussion be sure participants understand
that:
- Monitoring is an integral part of the evaluation
process. Through monitoring we gather ongoing information about project progress
using pre-established benchmarks or milestones; we periodically ask the
question, "Have we gotten as far as we had expected to get at this time?".
- Two types of evaluation need to be done, Process Evaluation and
Outcome Evaluation. Process Evaluation, periodically looks at our
strategies and the activities devised to implement the strategies and asks the
questions: "Are we following the strategy we said we would follow or are we
doing something else? If we are following our strategies, what is our
assessment of their adequacy, effectiveness,
appropriateness, efficiency?" In Outcome Evaluation. we
look at our objective, at the predetermined time, and ask the question "Did we
accomplish what we set out to do?.
|
Step 4 (35 min)
Developing Evaluation Plans
Ask the group to form the same work pairs as in Session 20 and
explain that their task for the next 30 minutes is to develop a plan for
evaluating the health education projects they have elaborated an their Project
Planning Worksheet. Because of the limited time, tell them to focus on only one
of their strategies and related activities.
Tell them to reevaluate and, if necessary, modify their objectives
and any other items they have listed on their Project Planning Worksheets. Ask
them to attach the evaluation worksheet to the back of their Health Education
Planning Worksheet when they have completed it, and to be prepared to describe
their plans to the group in the next step.
|
Trainer Note
While the group is working in pairs, walk around the room to
observe how the process is proceeding and assist any groups that seem to be
having problems. Also refer them to pp. 79 of Handout 20A (Planning a Community
Health Project) for further assistance or guidance in this
Step.
|
Step 5 (25 min)
Presenting and Discussing Evaluation
Plans
Reconvene the large group and ask for two volunteers to briefly
describe their evaluation plans. After each presentation, ask the rest of the
group to offer comments on the plan and suggest ways to improve it.
When discussing each plan the following questions should be asked:
- Did you base the objective on information you
collected on assessing the needs or knowledge, attitudes and practices of the
community (i.e., do you have baseline information to evaluate the change your
program is to create)?
- Are your strategies and related activities adequate,
appropriate, effective, efficient?
- Are you evaluating how well the program is being done (i.e.,
process) as well as the outcome after the time specified in your objective?
- Are you including the appropriate people in your evaluation?
- Are you conducting the evaluation at the appropriate times?
- Will the results you think you might obtain be helpful to you in
determining whether to continue the program as is, change it or terminate
it?
Close this session by emphasizing the importance of basing
evaluation on objectives and clearly planning from the beginning how and when it
will be
done.
