CHAPTER 12: Assessment methods

The previous chapter discussed the general issues related to the assessment of students. This chapter describes specific methods that will help to improve the way you assess your students. Examples of each method are given and their advantages and disadvantages are discussed.

12.1 Oral examinations

In an oral examination, each student is interviewed by one or two examiners. Usually students are asked to tell the examiner what they know about some topic or what they would do in some situation that might happen in their job.

The main advantages of oral examinations are that the examiner can ask for more detailed information and can probe to find out how much each student knows.

However, this is not a very satisfactory method of assessment. Students are often made extremely anxious by examiners, even though the examiners try to be friendly. This is unfair on the students, because they will not face this kind of stress in their job. Many students get worse results in oral examinations than they deserve. Oral examinations also take up a lot of time and are often criticized because the marking is unreliable. Further, oral examinations rarely test the important skills and do not usually help students to learn.

You should not use oral examinations to assess students unless you have some specific reason for doing so.

12.2 Essays

Essays have been widely used in assessing students in the health professions. But again, this method has very serious disadvantages.

In one course, students were asked to write an essay on polio immunization. This is a very poor test even though the topic was of some relevance to the students. (The students were going to be responsible for immunizing people against polio as part of their jobs.) The test is poor because:

· The students cannot know what information the examiner considers to be important. For example, should they describe the administration of an immunization programme? Should they outline how immunization prevents polio? Or should they describe the side-effects?

· The marking is likely to be unreliable. Because the topic is not clearly defined, different teachers will think different points are more important - and give different marks as a result. Whether a student passes will depend very much on who marks the paper.

· The test is not valid. Students are not going to write essays in their job. They are going to immunize people. Therefore it would be much better to test the skills required for this task.

· The essays will take a long time to mark - if teachers do this job thoroughly.

· The students are unlikely to learn very much from the test.

How could the essay be improved?

The first point is that a different method of assessment would probably be better - some examples are described in the following paragraphs. However, if an essay must be used, you should:

1. Make the title much more specific, for example:

"Describe how you would explain to mothers why their children should be immunized against polio" or "Explain how polio vaccine should be transported and given to children".

These essay titles are fairer because it is more clear to students what they should write about. They are also more valid because they ask students to describe the skills that are important.

2. Prepare a marking scheme and follow it. This scheme should include a list of the major points that should be covered in the essay and specify how many marks should be given for accurate spelling, general clarity of explanation, etc. The scheme should be used by all teachers marking the essay. This improves reliability.

3. After the examination, show the marking scheme to the students and discuss it with them. This helps them to learn.

12.3 Short-answer questions

Short-answer questions allow teachers to ask questions about a larger proportion of the course and to mark more accurately and quickly.

Example of short-answer questions

The following questions were part of an examination for trainee health inspectors.

1. List four advantages to a household of proper rubbish disposal.

(i)
(ii)
(iii)
(iv)

2. Draw a diagram showing the construction of a simple incinerator suitable for use in a small village.

3. Give two examples of situations when burying rubbish is better than composting.

(i)
(ii)

Short-answer questions often ask students to give examples, write down some advantages or draw a diagram. Because they are so much more specific than essays, they are quicker to mark and more reliable. They are also very much quicker to answer. This means that the students can be tested on many more topics during the examination.

The main disadvantage of this method is that it may simply test the students' ability to remember facts rather than apply knowledge or use skills.

12.4 Multiple-choice questions

Multiple-choice questions are often called MCQs. They are a stage beyond short-answer questions, because the students do not write any words. They just choose which of several answers is best.

Although you can use four or six choices in multiple-choice questions, five is the most common number. This type of question is sometimes called the "one-from-five" type of multiple-choice question.

Example of an MCQ of the one-from-five type

A patient tells you that he is worried because one of his eyes is red. You cannot find any foreign bodies in the eye, but note that the pupil is bigger and does not respond to light. What is the most likely diagnosis?

A. Trachoma
B. Conjunctivitis
C. Iritis
D. Corneal ulcer
E. Glaucoma

In this example, the student has to choose between the possible answers and select the best one - in this case "E". In this type of question, there is a stem - "A patient tells you... likely diagnosis" - and five choices.

Another type of MCQ is the true/false type.

Example of a true/false MCQ in glaucoma

A. There are usually white or gray spots on the cornea.	T. F.
B. The pupils are irregular.	T. F.
C. Only one eye may be red.	T. F.
D. The patient should be referred to a health centre.	T. F.
E. A foreign body is the most likely cause.	T. F.

Again there is a stem - in this example it is very short - "In glaucoma".

But this time the stem is followed by several statements. For each statement the student has to decide whether the statement is true or false. In this case "A" is false, so the student should draw a circle round "F". "B" is also false, but "C" and "D" are true while "E" is false, so the student should draw circles round F. F. T. T. and F respectively. In this example the student has to answer all five parts of the question.

Both these types of questions are used fairly commonly, although true/false questions are often preferred because they are easier to understand and can be used to test the students on a wider range of facts.

How useful are MCQs?

MCQs can be marked very quickly and accurately. They can also be answered quickly, so a lot of questions can be set in an examination. This means that a lot of the course can be covered.

On the other hand, there are serious disadvantages. It is quite difficult to write clear questions - so writing the questions takes a lot of time. There is also the very serious problem that MCQs usually only test the students' knowledge. Only rarely do they test decision-making skills and they cannot test the students' ability to communicate or to perform procedures. This means that MCQs are likely to be valid for only a small part of your course.

Despite these problems, MCQs can be useful. They can be used to check factual knowledge, especially during the course. They are also very helpful when used for self-assessment or peer-assessment.

If you decide to use MCQs, the following points may be helpful:

· You should allow roughly 2 minutes for each 5-part true/false question in an examination. So in an hour students can be expected to answer about 30 questions. If you find that students are not finishing the examination, cut down the number of questions. It is not a race.

· For true/false questions, give the students one mark for each correct choice, zero for no answer and take away one mark for each wrong choice.

Use the same scheme for marking one-from-five questions, but do not take away marks for wrong answers.

· The pass mark for MCQs should be quite high. This is because the MCQs should be testing basic knowledge that all students should know. Therefore a pass mark of 80% or 90% can be used successfully. It is better to use easy questions with a high pass mark rather than harder questions with a pass mark of 50% or 60%.

· Marking is made much faster if a separate response sheet is used for the students' answers. Then a mask can be laid over the response sheet, with holes cut out for the correct answers.

Look at the example below. Three correct answers will show through the holes - so you would give 3 marks. There are four ticks altogether, so one answer must be wrong - so you would take away one mark. This leaves a score of 2 (3 - 1) for the student.

Figure

12.5 Patient-management problems

Patient-management problems are a development of short-answer questions. The main feature is that a series of questions are asked about a case. This method can be used to test students on a wide range of subjects. It can be used wherever students are being trained to make decisions. So it is very useful for assessing students who are training to be health educators, community health workers, community nurses, health inspectors, etc.

Example of a patient-management problem

Mrs A comes to the health centre and tells you that she feels tired all the time. She asks you for a tonic. You find out that she is 30 years old and about 5 months' pregnant.

1. List 3 things that you think might cause the tiredness.

2. Write down 2 other questions that you would like to ask Mrs A.

3. As a result of Mrs A's answers, you suspect she is anaemic. What physical signs would you look for?

4. Your examination confirms your diagnosis of anaemia. What treatment (if any) would you prescribe and what other advice would you give?

This example has the advantages of a short-answer question. It is clear to the student what is required and it will be quick and reliable to mark (providing that all teachers involved agree what the possible causes of tiredness are). It is also much more valid as a test because it is based on the kind of work the students are being trained to do. (It would, of course, be much better if each student met Mrs A and took a history and examined her.) If students are given the marking scheme after the examination, they will also be able to learn from this.

How can you prepare patient-management problems?

It is usually easiest if you base the problem on a case that you have dealt with, such as a boy who presented with severe abdominal pain or a mother who rejected any advice on nutrition, even though her children were malnourished. Of course you can only do this if you still work as a health worker yourself. However, if you teach full time, talk to health workers or, even better, spend half a day with a health worker to write down examples of cases.

The next stage is to divide the case into stages. What happened first? What decisions had to be made? What alternatives were available?

Then you need to decide what items of information to give the students and what questions to ask them.

At this stage you will have a patient-management problem, but you will still need to develop a marking scheme. List all the answers that you think students might give - both right and wrong. Then decide how many marks you will give for each of the possible answers.

12.6 Project reports

In a number of courses, students are asked to work on a project. This may involve such tasks as doing a survey of a community or working in a health care team for a few weeks. Often the students then present a report on the project and this can take a lot of time.
Naturally students will be more motivated in the project if their reports are assessed and the marks count towards the final examination score.

However, project reports are extremely difficult to mark fairly because there are usually no clear standards for teachers to follow. Some students may do very good work but present a poor report. Others will present a very clear and full report of poor work. Which is best and what standard should you accept?

The following guidelines may help you to assess project work.

1. Project work should be assessed by at least two people marking independently. The two marks should then be compared and discussed to reach a final mark.

2. Where possible, explain to students what standards they should aim for. Tell the students what you think a good project would be like. Where possible, explain how much data should be collected, how many cases should be seen, and what kind of graphs or tables would be useful. But be careful not to restrict the students too much.

3. Show the students some project work done in previous years that you think is good and also some that you think is bad. Explain your reasons. Of course you cannot do this when you first use projects - so it may be better not to count the marks for the first projects in the overall assessment.

Clearly, the use of projects in assessment causes some problems for the teacher. What is their value? Project reports take a lot of time to mark and the score may not be reliable. However, this method of assessment can be valid if the projects are chosen carefully to involve the students in practicing the important skills. Above all, projects can be very powerful learning experiences and they should be assessed to encourage students to make the maximum effort.

12.7 Casebooks

Casebooks have been used quite widely in training nurses and they can also be used in courses for other groups of primary health care staff.

The casebook contains a list of skills or tasks that the students should be able to do. These tasks are the objectives, or at least some of the objectives, for the course. The students are responsible for learning how to do each of the tasks, and when they are ready they can ask a teacher to assess their performance. During the course the students must perform all of the tasks to a satisfactory standard. If the teacher thinks that a student's performance is not good enough, the faults are explained and the student can try again later.

Example - A page from a student's casebook

Task	Date	Signature
17. Prepare a flip chart for use with an audience of 30 people
18. Give advice to a pregnant woman about antenatal care	20/10/90	Unintelligible

This assessment method uses quite a lot of the teacher's time because each student must be seen and his or her performance must be assessed. It can be difficult to organize because teachers may not be available when the student is ready to be assessed. There may be problems with reliability. On the other hand, there are a number of advantages. The main advantage is that casebooks help learning. They do this by making clear to the students what needs to be learned. They also make sure that when students are not up to standard, the teacher is there to give advice. The second advantage is that the method is highly valid, because students are assessed on how well they can do the tasks and jobs that they are being trained to do.

This is a slightly different type of assessment. You do not give students a mark out of ten for each performance - you simply decide whether they are good enough or not. So at the end of the course, a student may have performed 23 out of the 29 set tasks to a satisfactory standard. It is then up to the examiners to decide whether this is a "pass". For some courses, students will need to achieve a satisfactory standard on all the tasks. For others, it may be unnecessary to insist on this high standard.

12.8 Checklists

Checklists are not so much a method of assessment as a way of improving other forms of assessment. Assessments of practical and clinical work are often criticized because the mark is unreliable. Different examiners use different standards. Checklists reduce this problem and they also make sure that the way in which the student does the task is assessed.

Example - A checklist for the task "Prepare a thin blood film using a sample of your own blood" not done done correctly

1. Uses middle finger or ring finger

2. Cleans the finger using surgical spirit or alcohol

3. Dries finger with a clean piece of cotton wool

4. Allows blood to flow freely after pricking with needle

5. Puts a single drop of blood in the middle of the microscope slide

6. Uses a second slide as a spreader. Allows the blood to spread along the end of the second slide

7. Pushes spreader quickly along the slide

8. Draws blood along behind the spreader

9. Does not blow on slide or shake it

The examiner can watch the student preparing the blood film and put a tick in the right-hand column for each part done correctly. At the end of the test, the examiner adds up the number of ticks in the right-hand column and gives the student a score out of 9. The pass mark for this test needs to be decided by the examiner. In this example, the examiner may feel that 7 out of 9 would be a suitable pass standard. For other tests, the examiner might expect 50% or 90%. The pass mark will depend on the specific test.

The advantage of a checklist is that it makes the marking fairer. Different examiners watching a student do a test are more likely to give the same score if they have a checklist. Checklists are also very useful for giving feedback to students or teachers because the evidence is clear and simple. The examiner might tell the teacher "Most of your students did the blood film test quite well, but I noticed that about half of them pushed the drop of blood instead of drawing it behind the spreader slide". This would help the teacher realize that this point needed more emphasis during the next course.

In the same way, detailed information can be given to each student. For example, students might be allowed to see the checklist for their own performance.

The above example is for assessing a psychomotor skill. Similar checklists can be prepared for assessing communication skills and attitudes, but this is often rather more difficult.

Note that you can prepare a checklist from a task analysis.

12.9 In-course assessment

During the training course, your students will probably spend time working in hospitals, health centres or dispensaries. There they will be practicing the communication skills and the psychomotor skills needed in their job. This time can be used for assessment as well as teaching.

Many different people will assess the students, so in-course assessment is likely to be more reliable if supervisors are given a checklist to follow. This checklist should be fairly simple, as shown in the example below.

Example - A checklist for assessing students in a health centre

completely	just good	not good
satisfactory	enough	enough

1. Keeps complete and accurate records
2. Observes sterile procedures
3. Establishes good relationships with patients and so on

Nurses or health workers supervising students can use simple checklists to give a clear picture of what the students can or cannot do. You can then use this information:

1. To make decisions on whether students should pass or fail.
2. To give specific advice to students about what they need to learn.
3. To improve the course in areas that are poorly learned.

This kind of checklist is again prepared from a task analysis. Checklists can also be used to help assess attitudes.

Example - A checklist for assessing attitudes

1. Very keen willing worker	|—|—|—|—|—|—|	Does as little work as possible
2 Accepts instructions	|—|—|—|—|—|—|	Resents or ignores instructions
3. Very interested in patients	|—|—|—|—|—|—|	Not interested in patients
4. Always keen to learn	|—|—|—|—|—|—|	Not interested in learning
5. Always on time	|—|—|—|—|—|—|	Always late

This checklist might be used by a senior nursing officer or supervisor on a ward where student nurses spend part of their training. The supervisor would use one form for each student nurse. At the end of the training period, the supervisor would think about the way each of the nurses had worked during their time on the ward.

For example, some nurses might have been quite willing to do what they were asked to do, but never seemed very keen or offered to do extra work. The supervisor would note this down on their forms by putting a cross at about the middle of the line:

1. Very keen willing worker

|—|—X—|—|—|—|

Does as little work as possible

In this way the supervisor can give a fair and quick summary of the attitudes of the student nurses to the teacher responsible for the course. This checklist can be used to give advice to student nurses and can form part of the overall assessment that is used to decide whether they pass or fail the course.

12.10 Summary

No assessment method is perfect. Each has some advantages and some disadvantages. You should therefore use a variety of methods whenever possible.

Ideally, you should first decide what skills need to be assessed. These skills are the performance objectives of the course.
Then you should choose the best method for assessing these skills. The method should be chosen on the basis of:

- regulations for the course
- economy of time
- reliability
- validity
- value as a learning tool.