Handout 21A: Field monitoring and evaluation of communication campaign

Field Monitoring of Communication Campaigns Once an information campaign has been launched, almost immediately questions such as the following begin to arise. "Is it going as well as hoped?", "Is the campaign weak or failing in some respects?", "What parts of it are weak and what parts are strong?", "What adjustments are needed?" Those who are sponsoring and paying for the costs of the programme do not want to wait until the campaign has run its full course to get answers to these questions. They want (and deserve) answers to these questions after the campaign has been in operation for only a few days. Factual evidence that can save time and money and prevent failures (or make success even more complete) is needed urgently at the mid-stream stage of a communication campaign.

"Field monitoring" or "midstream evaluation" is a special subtype of communication research which has the capability of providing answers to these questions. It is not a single type of research activity, but a rapid assembly of bits and pieces of evidence and data which, when put together, will provide an approximately accurate picture of how the intended audience is responding to the campaign. It is not necessarily something which is done only once during the course of the campaign. It may be desirable to take monitoring "soundings" at weekly, biweekly, monthly or quarterly intervals. The amount of detail, the frequency of taking of soundings, and the amount and type of analysis performed must depend upon the nature of the project and the context in which it is being performed. The notes which follow are intended to be a general guide, which must be amplified and modified to fit individual projects.

In order to be useful and effective, each episode or sounding by field monitoring must have the following characteristics

1. It must be done quickly-the information is needed now;

2. It must be done cheaply-most communication campaigns have zero budget for monitoring;

3. It must focus on a few specific questions of critical import to the success of the project;

4. It must supply reliable and valid date within the limits of tolerance of accuracy required by the decisions to be made.

Devising a monitoring procedure which conforms to the above specifications is not as difficult as might seem at first. Luckily for most programmes, statistics or data which are correct within 10 or 15 per cent of the true measurements will be adequate for making the decisions that need to be made concerning the need to change the programme, or to continue with the original plans.

"Hard" versus "soft" data. One controversy which arises immediately in discussions of monitoring, is whether or not it is sufficient to interview informally a few respondents and write or make a verbal report on the general impressions gained from this exercise. "Soft" research of this type is all that is needed, some will contend. Others will insist that only collecting "hard" statistical data from a fully representative sample of the intended audience will be sufficient. Proponents of "soft" data will insist that check-lists and multiple-choice questions are inadequate (even though they make possible impressive statistics), because monitoring requires that people be allowed to criticise the messages in an uninhibited way—using words of their own choosing. The position taken here is that both of the above philosophies are correct. Good monitoring methodology involves taking the best ideas and techniques of each and combining them into a research procedure which will produce hard data where hard data is essential and soft data where soft data is essential. It also involves avoiding the pitfalls and weaknesses of each of these methodologies as they are usually practiced. Because "soft" data often interviews a biased sample of respondents' taken from only one single site, there is a grave danger of overlooking the wide variation in response which the campaign is revoking. Monitoring must take observations from a number of different sites, and it must develop some system for making sure that the respondents at each site arc not selected because they arc easily accessible, co-operative, or some other reason which might bias their reaction. Because "hard data" methods make it impossible for people to express their thoughts in their own words, good monitoring procedure calls for the generous use of open-ended questions which encourage the expression of frank opinions in an uninhibited fashion. Techniques of content analysis can be used to code these answers.

Sample size

Like all research, the precision of the monitoring soundings will depend primarily upon the size of the sample drawn and the procedures by which respondents are selected. Taking huge samples by highly precise sampling procedures consumes much time and drives up costs. Inasmuch as most of the data must come from a mini-survey, carried out on a small sample of respondents within a very short time, the sample size will be a primary consideration. In general, a rapid survey of 50 cases, taken from 10 different localities widely distributed and varying in characteristics across the entire range of communities (five interviews being taken at each point) will provide sufficiently exact evidence to let the sponsor know whether he is succeeding or failing-and by how much and in what respects and why With a short questionnaire and with such a small sample, it is possible to conform to the four key specifications prescribed in the opening statement. If greater precision is required, a larger sample involving more sampling points can easily be planned—if time, manpower and funds are available.

Cost and timetable

If a 50-case mini-survey is prescribed as the major source of monitoring data, the ultimate costs and timetable can be estimated within a fairly small margin of error.

(a) Interviewing time
With a short questionnaire and reasonably good transport, one interviewer should be able to complete five interviews in one day. For 50 interviews a total of to person days will be required. If a total of 5 interviewers are used, the entire interviewing procedure can be completed within 2 days. (In order to avoid interviewer bias, it is recommended that not less than 5 interviewers be used.)

(b) Coding of interviews
The "hard data" items on the interview will be self coded (pre-coded). The "soft data" (open-ceded) items will require only very broad content-analysis coding. One person c" code the 50 interviews ID 2 days. Coding can begin as quickly as interviews begin arriving from the field, so that the completion of coding can be achieved within one half-day after completion of interviewing.

Tabulation

The statistics needed to generate "hard data" are usually only simple frequency counts or cross-tabulations. The 50 cases can be hand-tallied by one person working one day. If a counting sorter is available, it will be faster and more reliable to punch cards and make the tabulations by machine—within an hour or less. But machine tabulation is not important unless some fine points are to be explored using sophisticated statistical procedures. (Some of the most powerful and complicated statistical procedures involve drawing useful conclusions from very small samples of data. Therefore, if a computer is available, it is possible to follow up the first preliminary monitoring with a more complete and precise one.)

Total timetable

A complete monitoring exercise can be carried out within one week as follows:

Monday (first day). Finalize the questionnaire, duplicate the questionnaire and instruct the interviewers. Give out field assignments.

Tuesday (second day). Begin field work. Interviewers must be transported to the sample sites and must be transferred from interview to interview quickly. Ideally each interviewer should be driving his or her own vehicle in order to minimize time consumed.

Wednesday (third day). Complete the interviewing. If interviewing lagged the first day add additional interviewers or arrange to work longer hours to complete the task. Begin the coding. As each interview is coded it can be handed over to the hand tabulator for tabulation.

Thursday (fourth day). Complete the coding and hand tabulation. As quickly as the last interview is coded and tabulated the data should be handed over to the analyst. The analyst meanwhile has been studying the preliminary results of the hand tabulation as they are made so that he already knows what the major findings will be. He begins writing his report. By working in the evenings he finishes.

Friday (fifth day). The report need be only a few pages with a few simple statistical summaries and tables. It can be completely typed (the tables can be duplicated copies of the worksheets) by noon. The afternoon of the fifth day can be devoted to a presentation of the findings to the programme director and a general discussion of what implications they have for the campaign. Decisions for change and revision can begin to be reached.

In most cases there will be adequate time to carry out these steps at a somewhat slower pace. The above timetable is presented only to show the spirit and speed which is possible in good monitoring research.

Total cost of monitoring

The total cost of one episode of monitoring is less than $1,000 US. The cost at overseas sites should be considerably less. The following is the breakdown of costs for a 50-case mini-survey:

Interviewers—10 person days at $25 per day	$250
Typist—2 person days to type questionnaire, report	50
Coder end hand-tabulator—3 person days combined	75
Supervision of interviewing and coding	100
Transportation of interviewers	200
Salary for analyst study director	150
Incidental expenses	50
Total cost of monitoring "sounding"	$875

The above costs cover all expenses. If the manpower is obtained simply by transferring to the project personnel which are already on the payroll, the out of-pocket costs are almost zero— involving only the transportation and incidental expenses.

Inasmuch as most communication campaigns run to many thousands of dollars, the expenditure of the above amount of funds could be a most valuable investment. The modest cost makes it easier to justify two or more rounds of monitoring if the campaign is a prolonged one.

The Field Monitoring Interview

One secret of effective monitoring is to limit the enquiry only to a few questions that are of crucial importance for guiding the campaign. This is no time for the researcher to exercise his general scientific curiosity. Instead he must justify every item placed on the questionnaire in terms of how directly it is related to questions that have been raised about the adequacy of the campaign. In most-eases this will limit the questions to five categories of items:

1. Is the intended audience receiving the message ?

2. Is the intended audience comprehending the message and remembering its contents?

3. Does the intended audience accept or reject the content of the message?

4. Is the aver-all reaction of the audience toward the campaign favourable or unfavourable? What do they find offensive?

5. Is the intended audience responding to the message by changing its behaviour in ways desired by the campaign directors?

Good monitoring interviews will contain questions which will obtain valid data on each of the above points. It will be confined to these categories of questions plus a few "demographic" questions concerning the basic characteristics of the respondents needed to interpret the results.

In order to illustrate a typical monitoring questionnaire, a crude prototype has been developed and presented as Appendix A. The exact wording of the questions would be done in terms of the local language and dialect. The wording given is intended only to convey the meaning. An example from nutrition is used, but the topic could be any one of a wide variety of programmes which are the subject of development communication.

The monitoring interview is so short that it can be completed in 20 minutes or less in most cases. Allowing 10 minutes for general rapport-building conversation and close-off, a total of 2.5 hours of actual interviewing time would be required per day to obtain 5 interviews. The remaining 5.5 hours could be spent in travel between interviewers. This gives a generous one-hour travel time between interviews.

Analysis of the Data

The responses to the questionnaire can be tabulated to fill in the following data-sheet.

1. Percentage of respondents receiving messages via each medium:

	Per cent
Home visits	_____
Group discussions	_____
Radio programming	_____
posters	_____
Leaflets or printed material	_____
Other media	_____
Received no message, any medium	_____
Received message via personal contact only	_____
Received message via mass media only	_____
Received message via both mass media and personal contact	_____

2. Percentage able to recall content messages. (Develop score):

	Per cent
High content recall	_____
Medium content recall	_____
Low content recall	_____
Received message, can recall nothing of content	_____
Did not receive message at all	_____

3. Percentage liking message presentation (by those receiving):

	Per cent
Home visits	_____
Group discussions	_____
Radio programming	_____
Posters	_____
Leaflets	_____

4. Percentage rejecting the message (by those receiving):

	Per cent
Home visits	_____
Group discussions	_____
Radio Programming	_____
Posters	_____
Leaflets	_____

5. Knowledge level of respondents, by receipt of message:

Level of knowledge	Received no message	Received message via
		Mass media only	Personal contact only	Both mass & pers.
High
Medium
Low

6. Change in feeding and nutrition practices:

Yes
No

7. Talked to others about nutrition:

Yes
No

If time permits, a sheet of the above tabulations can be prepared separately for male and female respondents, for old and young, educated and uneducated, for race or ethnic group, for income groups, etc. In every case the division must be a simple dichotomy because of the small number of cases. If a tabulating machine is available, the preparation of these supplementary cross-tabulations is made easier.

Content analysis of the verbatim responses to open-ended questions

No effort should be made to perform a statistical analysis of the verbatim responses to the open-ended questions in addition to that called for by the above summary. Instead, a typist should rapidly transcribe on a single set of pages the responses to each question, separately. These responses should be read and discussed in the evaluation, much as they would be in any interview which had relied on the "soft" data-collection approach. These specific comments can be invaluable in understanding and interpreting the reaction to the survey—what is wrong and what is going well.

Advance planning

In order to be effective, the field monitoring research must be planned well before the communication campaign begins. Because there will be need for rapid action, the questionnaire should be drawn up and pre-tested beforehand. The interviewers should be identified and instructed beforehand. The sampling sites and the plan for transporting interviewers to them should be drawn up beforehand. The coding and tabulation plan should be drawn up and tried out beforehand. In other words, there should have been a complete dress-rehearsal of the monitoring step before the communication campaign is launched. If this is done, all of the difficulties and delays that will be experienced will have been discovered and corrected. If the researchers wait until the campaign is already under way before they begin to develop a questionnaire and work up the research plans, the monitoring research will encounter many delays, there is a danger that the data will be poor, and that it will arrive too late to be of an) practical use. Advance planning is, therefore, absolutely mandatory.

Appendix A - A Prototype Interview for Field Monitoring of a Communication Campaign

A. Is the audience receiving the message?

1. Have you recently seen or heard or talked to anyone about (nutrition)?

Yes . . . . . . . . . No . . . . . . . . .

Ask following of everyone, irrespective of answer to above question:

a) Did anyone come to visit you in your home to talk about it?
b) Did you join in a group discussion where this was discussed?
c) Did you hear anything about this subject on the radio?
d) Did you see a poster or hoarding about it?
e) Did you see a leaflet or booklet about it?
f) Did you (see) (hear) (talk)...other media used in the campaign?
g) Did you talk about this with a friend or neighbour?

B. Is the intended audience comprehending the message and remembering its content?

Questions 2, 3, and 4 are to be asked for each item in Question I for which a "Yes" answer was obtained.

2. Will you please tell me as much as you can remember of what v as said about (nutrition) in the (medium)? Record the response word for word as nearly as possible, for content analysis.

Probes

a) Did they want you to try out a new thing? What did they want you to do?

b) Did they want you to stop doing something you have been doing? What did they want you to stop doing?

c) What reason did they give for wanting you to do these things?

C. Is the reaction of the audience favourable or unfavourable?

3. What was your feeling about this (medium)? Did you like or dislike (hearing) (seeing) (participating in) it?

If disliked: What was wrong with the (medium)? What caused you to dislike it? Record word for word.

D. Does the audience accept or reject the content of the message?

4. Did you think the information and advice given to you was correct or incorrect? Was it partly correct and partly incorrect?

If incorrect (all or part): What did they tell you that you know is not right?

Probe
What did they say that you know is incorrect? Record word for word.

E. Is the intended audience responding to the message by gaining knowledge and changing its behaviour?

5. Develop and insert a four-item knowledge test.

Example:

a) When a baby is weaned, is it better to feed the baby mostly (starch diet) or to feed it a mixture of (balanced diet)?

b) Suppose you were preparing a meal for children. Which of the following meals would be better to make them grow strong and healthy: a meal comprised of (traditional meal) or a meal comprised of (recommended balanced diet)?

c) Have you ever heard of "protein"?

Yes . . . . . . . . . No . . . . . . . . .

Which of the following foods has more protein?

Maize or beans?
Bread or cheese?
Carrots or chicken?
Oranges or bananas?

Which of the above foods is better for helping children grow strong muscles?

d) Have you ever heard of "vitamins"?

Yes . . . . . . . . . No . . . . . . . . .

Which of the following foods has more vitamins?

Oranges or maize?
Bananas or carrots?
Cabbage or beans?
Beef or bread?

Which of the above foods is better for helping children from getting sick?

F. Is the intended audience changing its behaviour?

6. Since you heard these message about (nutrition) have you changed the way you feed your children?

Yes . . . . . . . . . No . . . . . . . . .

If "Yes": What changes have you made?
If "No": What has kept you from making any changes?

a) Was it because you did not believe the messages?
b) Was it because you did not think it very important?
c) Was it because you could not get the foods you needed to make the change?

7. Since you heard these messages about (nutrition) have you talked with any neighbours or friends about (nutrition)?

Yes . . . . . . . . . No . . . . . . . . .

a) how many persons have you talked to? Who?

b) Do most of these people believe in eating mostly (starch diet) or do they believe in eating a mixture of (balanced diet)?

8. Characteristics of respondent:

a) Sex ... Male/Female.
b) How old were you on your last birthday? (Estimate to nearest 5 years if age not known.)
c) How far did you go in school? Husband ........... Wife ...........
d) Occupation of husband
e) Measure of income or wealth
f) Ethnicity or race
g) Number of children in family

(From: COMMUNICATIONS FOR SOCIAL DEVELOPMENT IN AFRICA, A Report of a UNICEF sponsored International Workshop held in Arusha, Tanzania, Dec. 1976.)