Introduction

Monitoring and evaluation determine how well an emergency preparedness programme is being developed and implemented and what needs to be done to improve it. The method can be applied to:

- developing and implementing policy;
- vulnerability assessment;
- emergency planning;
- organizational preparedness;
- training and education.

Three ways of monitoring and evaluating preparedness are described here:

- project management;
- checklists;
- exercises.

Project management

The means of monitoring and evaluating during the implementation phase of a project include: measuring the progress toward project objectives; performing an analysis to find the cause of deviations in the project; and determining corrective actions. (See Annex 1 for more details.)

Projects involve analysing the present and past, predicting the future, making changes, and developing new ideas and products for future use. Very often the analyses, predictions, changes, and new ideas and products are not entirely correct, and over time the environment in which the project is being implemented will change. In each part of the emergency preparedness process described in this manual it is possible to make mistakes, and there is always room for improvement.

Policies describe long-term goals and assign responsibilities, and may establish work practices and decision criteria. It is possible, however, that a policy goal may be set too high to be achieved or be incorrect in other ways. Policy review cannot be continuous, or the basis for all emergency management programmes would be continually altering and individual projects would not be completed. Policy-makers should remain receptive to criticism and suggestions, and should periodically review policies in the light of experience, changes in the policy and emergency management environment, and new challenges that arise, remembering that policy development is a participatory process. If a policy is embodied in legislation, a common reaction to suggestions for change is “But we can’t, it’s the law!” Laws are made to be useful and can be changed when they no longer serve their purpose.

Vulnerability assessment can determine community vulnerabilities, describe hazards and the harm they may cause, and provide information for all aspects of emergency management. The accuracy of a vulnerability assessment is determined by the quality of:

- community participation;
- the information used;
- the resources applied;
- the assumptions about the community, the environment, and the hazards;
- the conceptual models.

Vulnerability assessment will never present a perfectly correct picture of vulnerability, hazards, and potential emergencies. When an emergency has occurred, it is often discovered that the models used to describe the behaviour of a hazard are incorrect. For example, actual floods rarely follow precisely the flood heights and time scales predicted. The models therefore need to be fine-tuned. Assumptions about community vulnerability sometimes prove unfounded, and predictions of community behaviour during emergencies are not always correct. Thus, the analysis of emergencies - even minor ones that cause little harm - can yield information that will make a vulnerability assessment more accurate.

There is also inevitable change in the community, environment, and vulnerability. Effective vulnerability reduction and emergency preparedness programmes will create changes for the better, and economic, environmental, and social influences may create changes for the worse. Thus, vulnerability assessment needs to be reviewed periodically.

Emergency planning is intended to protect the community and its environment, and to reduce uncertainty and confusion during emergencies. Sometimes emergency plans do not work. One of the most common reasons for this is that plans were developed in isolation and not communicated to the right people. Other reasons may include:

- poor communication (both technical and personal) during the emergency;

- lack of coordination of response work, leading to duplication, inefficiency, and ineffectiveness;

- lack of resources for the problems at hand.

After each emergency, an analysis of the events and actions that occurred is required. Each organization involved should hold debriefings, and then there should be a single debriefing for representatives of all organizations. A debriefing entails presenting facts of the emergency, describing the role that each person or organization played, and evaluating the actions taken. While debriefings are instructive for those who participate, they should also be documented and used to improve emergency planning.

Checklists

Checklists can be used to evaluate an existing emergency preparedness programme or to assist in developing a new programme. Checklists constitute a “closed set” in that they are not tools for developing new ideas or strategies. They can, however, form a compendium of current knowledge based on prior experience, and they are simple and easy to use.

Annex 3 contains a number of checklists for emergency preparedness, as well as for response and recovery. These checklists are not exhaustive, and can be added to as experience is gained or to suit the context of a community’s preparedness.¹

¹Further information on assessing health sector emergency preparedness can be found in Guidelines for assessing disaster preparedness in the health sector, Washington, DC, Pan American Health Organization, 1995.

Exercises

A common way of monitoring and evaluating parts of an emergency preparedness programme is through conducting exercises, which can be used to test aspects of:

- emergency plans;
- emergency procedures;
- training;
- communications, etc.

There are many different types of exercise, each suited to different purposes. The purpose of an exercise, and the aspect of emergency preparedness to be tested, must be carefully decided and fairly specific. An exercise should not be conducted with the purpose of testing an entire emergency plan or all aspects of training. Some specific purposes for exercises related to communications include:

- to test the communications procedures contained within an organization’s emergency procedures;

- to validate the interorganization communications covered in a plan;

- to test the call-out procedures within an organization;

- to validate the lines of command and control defined by a plan;

- to test the ability of organizations to establish and maintain emergency operations centres;

- to test the response times of organizations involved in a plan.

Some typical types of exercise include the following:

· Operational exercise, in which personnel and resources are deployed in a simulation of an emergency.

· Tabletop exercise, in which personnel are presented with an unfolding scenario, asked what actions would be required, and how the actions would be implemented.

· Syndicate exercise, in which personnel are divided into syndicates to discuss and consider a given scenario, and the syndicate planning and response decisions are then discussed in an open forum.

There are also a number of different ways of organizing, conducting, and reviewing exercises. One way is to go through the following steps.

· Determine need. Exercises can be expensive and time-consuming, and sometimes dangerous. There must be a clear need for the exercise, and it must be targeted appropriately. An exercise writing team should be formed to define and design the exercise.

· Define exercise. This involves determining:

- the aim, objectives, and scope of the exercise;

- type of exercise;

- the authority for its conduct;

- the performance standards that will be used to judge the degree of success of the exercise;

- organizations to be involved;

- resources and budget.

· Design exercise. Exercise design involves determining:

- appropriate exercise scenario(s);
- any special aids that may be required;
- timelines;
- exercise appointments;
- exercise control;
- safety requirements.

· Conduct exercise.

· Conduct exercise debriefing. The debriefing should be a meeting of those involved in the exercise to consider the degree of success in meeting the performance standards and in achieving the objectives;

· Validate exercise. This involves determining how plans, procedures, and training can be improved on the basis of the exercise results.

Selection of exercise writing team

Some of the criteria for selecting members of an exercise writing team include:

· At least one member should have some expertise in exercise writing.

· If a number of organizations are participating, each of the major organizations should be represented.

· Members should have experience in the areas to be tested or validated.

· The chairperson of the writing team should be from the lead organization.

Exercise appointments

To ensure effective exercise control, exercise control personnel should include:

- an exercise director;
- an exercise administrator;
- exercise umpires or directing staff;
- visitor or media liaison officer.

For operational exercises, the following appointments may also be necessary:

- damage control officers;
- safety officers;
- scenario coordinators.

Summary

· Monitoring and evaluation involves determining how well an emergency preparedness programme is being developed and implemented, and what needs to be done to improve it.

· Three ways of monitoring and evaluating preparedness are:

- project management;
- checklists;
- exercises.