|The Quality of Learning and Teaching in Developing Countries: Assessing Literacy and Numeracy in Malawi and Sri Lanka - Education Research Paper No. 41 (DFID, 2000, 90 p.)|
This chapter describes the process of implementing the research project. The following issues are discussed.
i. Setting up the research project
ii. The operational framework
Setting-up the research project
Establishing the project in Malawi and Sri Lanka
In Malawi, it was agreed that the Planning Section of the Ministry of Education be responsible for coordinating the Project and that the Centre for Educational Research and Training, based at Chancellors College, Zomba should be responsible for providing local counter-part research staff to support the technical aspects of the project.
In Sri Lanka, it was thought appropriate for the research project to be based at the National Institute of Education. Counterpart research staff were provided by the Institute and given that it was necessary to work in three languages - English, Sinhala and Tamil, these counterpart researchers played a vital role in the technical aspects of the project with the teachers involved in the study.
Selection of the localities for the study
Decisions about selecting the locality for the study in each country were left largely to the local research partners but were guided by the following considerations.
That the district/area be roughly representative of the school population on a continuum representing on the one hand, highly under-privileged rural areas to well developed urban areas on the other (Govinda & Varghese, 1993: 19).
In Malawi, discussions were held with selected Regional and District Education Officers and a number of districts was visited. It was decided that the Lilongwe urban and rural districts should be selected as these would provide a sample of schools which was representative of all types of schools in Malawi.
In Sri Lanka it was important to ensure that the geographical area chosen for study was broadly representative of an ethnically heterogeneous population.
The Kurunegala District in the North Western Province was proposed as that most representative of schools in Sri Lanka. Another important consideration was its accessibility and the extent to which local education authorities would be willing to co-operate. The Provincial Director of Education in the North-western Province was active in selecting the schools and later became a central figure in the research process. A number of schools in the district were visited.
Selection of schools for the study
As with the selection of the localities for the study, the selection of schools was left to the local research partners. The following guide was established for selecting schools.
That the schools represented a reasonable variety in terms of size, and had varying levels of human and material resources and other conditions characterising the context of primary education in the country (Govinda & Varghese, 1993: 22).
Ten schools were selected in each country. In Malawi five schools were selected from the Lilongwe Urban District and five schools from the Lilongwe Rural District. In Sri Lanka, of the ten schools selected, these were broadly representative of the types of schools (1AB, 1C, 2 & 3) and included two Tamil schools.
Selection of teachers
Thirty teachers were selected to participate in the study in each country. Because of the differences in the cycle of education and the organisation of schools in Malawi and Sri Lanka, the method of selection for teachers differed.
In Malawi, from the ten schools selected to participate in the study, teachers were chosen to represent each of the three sections of the primary school - the Infant section (Standard 1 and Standard 2), the Junior Primary section (Standards 3, 4 and 5) and the Senior Primary section (Standards 6, 7 and 8).
In Sri Lanka, the pattern of school organisation is 5 years of primary education, 3 years of lower secondary, 3 years of secondary and 2 years of upper secondary. There is no division between primary and secondary schools and the majority of schools have both primary and secondary grades. Further, English is introduced as a subject for the first time in Year 3.
It was decided to select only year 4 teachers. Thus, in each of the ten participating schools, two year four classroom teachers were selected. Further, the English teacher in each of the ten schools was selected.
The operational framework
The research was organised around the stages described in the operational framework outlined in chapter 1. To recap, the research was organised in five broad phases. These are:
i. Reviewing the learning requirements as stated in curricula, syllabi, and textbooks in each country
ii. Establishing standards and norms based on the professional experiences of teachers and an emerging literature on learning
iii. Developing procedures for assessment
iv. Administering assessment tasks, collecting evidence of childrens achievements and recording outcomes
v. Developing profiles of childrens achievements
These phases are described in more detail here.
1. Reviewing learning and teaching
The first phase of the research was to review the processes of teaching and learning in Malawi and Sri Lanka and principally, the expectations embedded in both systems as to what constituted successful learning. In other words, we needed to understand what the requirements of Primary Schooling were -and what learners were expected to achieve at different stages. Thus the Primary School Teaching Syllabuses for English, Chichewa and Mathematics for Standards 1 to 8 were reviewed in Malawi. These syllabuses were examined in terms of their subject objectives. In addition textbooks in use in Malawi primary schools in English Chichewa and Mathematics were collected and the range of activities examined.
Similarly, the syllabi for Year 4 in English and Mathematics were reviewed in Sri Lanka. At the time of the study, Sri Lanka had begun work on developing an Essential Learning Continuum (a criterion-referenced framework for the assessment of learning).
From these documents, an impression was gained of the demands the curricula made on students. The question that arose for both countries was whether the demands of the curricula were unrealistic in their expectations of students achievements. This assumption needed to be tested.
The processes of teaching were also scrutinised carefully. The teachers selected for the study were treated as focus groups and discussions about their professional practice offered useful insights into how learning was perceived and supported. A selected number of classroom observations were also carried out in particular to review the context in which Literacy and Mathematics were taught and learned.
In sum, the findings from these reviews were:
i. expectations of learning embedded in curriculum material, syllabi and textbooks were in the words of teachers ambitious, unrealistic, and unclear.
ii. The measures (tests and examinations) used to assess learning outcomes were too narrow and incapable of determining what learners know or can do.
iii. Approaches to supporting learning (teaching) were limited, mainly due to the lack of resources and the quality of teacher training and dated classroom methodologies.
2. Establishing standards and norms and developing assessment tasks
The second phase in each country was designed to work with teachers with a view to developing a framework to profile learning achievement. Teachers were asked to concentrate on a limited number of Learning areas. These were confined to Literacy (speaking, reading and writing) and Mathematics (number, measurement, area, investigations).
Meetings were held in both countries to review indicators of achievement in Literacy and Mathematics derived from the assessment systems of a wide range of countries. Teachers were asked to develop, from their own professional understandings of what learners are capable of achieving, a hypothetical framework consisting of four levels on which childrens learning could be profiled. Examples of how this was achieved in other countries were shared and discussed.
In both countries, teachers developed a rudimentary framework consisting of the following levels.
Basic knowledge, skills, competencies and attitudes
Developing or emerging skills and competencies
Skills and competencies firmly established
Advanced skills, knowledge
Work was also begun, in a very rudimentary way, on developing procedures to gather evidence of what children could do. This included thinking about ways in which teachers could use assessments in classes which were often as large as 120. The principle, was to involve teachers in the development of procedures for assessing students.
3. Developing procedures for assessment
Teachers began to develop an initial set of assessment tasks. It was necessary, however, for these tasks to be refined and developed farther. The initial set of Assessment Tasks produced by teachers at an in-country workshop were brought back to the University of Bristol for further development. (The completed set of tasks is contained in a separate volume). It suffices here to describe briefly the purposes and nature of the assessment tasks:
In Malawi the researchers worked with groups of teachers from the Infant, Junior Primary and Senior Primary sectors to develop tasks in various areas of the English and Mathematics curriculum. For the first trial activity in schools, the researchers developed tasks in the number curriculum at the three levels.
In Mathematics the purpose of the tasks was to provide data on what pupils knew and were able to do, on the difficulties they experienced by pupils and the mistakes which they made. To this end, three sets of tasks were developed at each level. First, there was a set of aural tasks where teachers would read questions to the class and the pupils would make individual responses on paper. Second, there was a set of tasks presented in written form for individual response by pupils. Finally, there was a group task which involved small groups of pupils working together and producing both written and oral reports on their work. Tasks were usually of a closed kind with a single correct answer the achievement of which required the pupil to possess specific from knowledge or skill, However, there were a number of less orthodox problems and more open questions which were intended to reveal pupil thinking.
The Languages tasks were divided into reading and writing and included three tasks for each of the different levels. For the Infant section, Task 1 was an early spelling task designed to assess whether children could recognise and use single letters and groups of letters to represent whole words or parts of words. The second task assessed childrens capabilities as independent writers, while the third task assessed early reading. Here, the purpose was to find out whether the child understood that it is the print (the letters and words) which tell the reader what to read.
For the Junior Primary section, three tasks were developed. Task 1 was designed to gain an understanding of childrens reading, particularly their independence, fluency, accuracy and comprehension. A reading record was created from a passage indigenous to the Malawian context and teachers were to use this record in gaining an impression of children as independent readers. Teachers were further required to make judgements about childrens comprehension by asking them to retell the content of the passage and by making predictions of what might happen next. The second task was designed to assess childrens capabilities as independent writers. This was divided into three types of task which were, an aural task where children listened to a passage read to them and they were asked to make up their own ending to the story. Task two required children to write a short story from a picture provided and Task 3 required the children to give a factual account which involved a description of their school.
The Senior Primary Section were provided with reading and writing tasks. The reading task required pupils to interpret tables and diagrams in order to test their comprehension and the writing task demanded the production of a personal account.
An accompanying set of Notes for Teachers was provided for each set of tasks.
In Sri Lanka, the procedures involved working with the National Institute of Education (NIE), rather than with teachers directly because of the language problems. The NIE staff were organised appropriately to include three members each of the Language Committees in Sinhala, Tamil, English and Mathematics. As in Malawi, the task was to establish a rudimentary framework for profiling learning.
The NIE staff engaged with the task of producing assessment tasks for Year 4 students. The orientation of the tasks was similar to those developed for Malawi. As was the case in Malawi, these tasks remained in fairly rudimentary form and were brought back to the University of Bristol for modification and further development. The tasks were to be returned to the NIE for translation into Sinhala and Tamil.
Further meetings were held in both countries to finalise the tasks for implementation and to set up the basis from which the Centre for Educational Research (CERT) in Malawi and the National Institute of Education (NIE) in Sri Lanka could provide support to the teachers in administering the tasks. Both CERT and NIE organised meetings with teachers and trialed the tasks in the classrooms.
Trials - Administering tasks, collecting evidence and refining the profiling system
A major activity between the second and third phase of the study involved trialing the tasks. Teachers administered the assessment tasks in their classrooms to establish the usability of the tasks and childrens responses to them. They also collected samples of childrens work. These samples were needed to refine the levels in the profiling framework.
Once the first set of tasks had been administered, teachers in both countries brought samples of childrens work to a meeting. In preparation for the meeting, teachers analysed the samples of work and awarded each piece of work a level. They wrote their justifications for awarding the different levels on the basis of answering questions about the piece of work. These questions were provided on the teacher record sheets for each task.
At the meeting teachers worked in pairs and small groups and discussed the samples of work they had brought with them. They exchanged samples and re-rated pieces of work, all the time providing justifications, on the basis of the work in front of them, as to the levels being awarded.
Through this process, teachers in each country and in each sub-group were eventually able to agree on samples of work which could be used as exemplars for each of the 4 levels in the profiling framework. These exemplars and other samples also provided teachers with the basis for developing statements to describe what might be expected at each level. These level descriptions were kept broad enough to accommodate most of the samples teachers had been exposed to over the course of the meetings.
4. Administering tasks, collecting evidence and developing profiles
The third stage of the research involved teachers and researchers in administering a second batch of tasks in their classrooms. This stage required the teachers to work independently to administer the tasks and to collect samples of childrens work.
Each sample of work was to be analysed by the individual teacher who was guided in doing so by questions included in the Teacher Record Sheets provided with each task. Teachers also had to write their own comments of the standard the work had achieved and to award a level score on that basis.
5. Profiling Learning
The final stage of the research involved teachers meeting together in order to standardise the assessment framework and to ensure that their judgements were reliable and valid.
Once again teachers selected samples of their pupils work which they felt spanned all four levels. Once again, they worked in pairs and small groups where they exchanged samples and validated the levels awarded to each piece of work.
When it was clear that the teachers assessments were standardised, i.e., that the reliability of their ratings was consistently high (inter-rater reliability was scored on Cohens Kappa) teachers were organised in groups to profile the entire sample of pupil scripts from all the sample schools.