Assessing whether an educational system is effective

The example of Djibouti
by Bernard Lebrun

Up to now, assessment of education systems has focused primarily on measuring their significance in the economy of a country, taking into account the number of successful pupils at a specific stage in their education in relation to the investments made. The quality of the teaching, in particular the educational yield is rarely examined. For the study presented below, it was necessary to devise appropriate assessment material, to choose representative samples and to maintain a strict approach to the experiment. The results fully reflect the effort involved and enable politicians to adopt practical measures to remedy deficiencies and safeguard achievements.

An assessment of the yield of the Djibouti education system had been planned as part of the Education II project financed by the World Bank. The objective consisted of assessing the standard of and cognitive yield from primary education and the first stage of secondary education, in French and mathematics, in an across-theboard study on the one hand, one year before the end of the stage, and in an up-down study on the other, comparing the results of the pupils one year later.

In primary education, the study started in 1992 with assessment of the performance of pupils in their fifth year of primary school. The following year, the data concerning the school and the class were collected. The pupils tested the previous year were retested in the sixth and final year of primary school. The test results were then compared against the marks given by the teachers in school reports and the results achieved by the pupils in the examinations for entry into the first year of secondary education. For the first stage of secondary education, the study began with an assessment of the performance of pupils in their third year. The pupils tested the previous year were retested in their fourth and final year of junior secondary education. The test results were compared against the marks given by the teachers in school reports and the results obtained by the pupils in the BEPC, the examination taken at the end of lower stage of secondary education. Information concerning the socio-economic characteristics of the pupils and their attitudes to the French language was collected from the pupils in the sample who took the tests in French.

The assessment was therefore conducted over two school years, in strict compliance with the scientific constraints. In the first stage of secondary education, different samples were taken for mathematics and French. A strict plan for the experiment was followed as regards both the procedures for devising the tests and the choice of samples. The precautions taken when taking tests, the evaluation of these and the recording of the results enabled high quality material to be obtained and properly used with the help of appropriate data processing techniques. The material gathered during these assessments could incidentally serve as a basis for future studies.

A summary dossier, setting out the entire experimental procedure and the most significant results for each of the two levels, was submitted to technical and decision-making committees, composed of educationalists and administrators, whose job was to examine them, extract the main findings from them and draw up reasoned proposals. The findings and proposals were forwarded to the political authorities responsible for education, whose task it is to adopt measures to remedy the deficiencies observed.

Quality and yield of teaching

The term 'yield' in the education sector, and especially in terms of measuring the quality of teaching, consists in comparing the performance of the pupils against an expected result at the end of a period of learning, determined by a 'leaving profile' of the pupil or, in the absence of this, by the content of the curricula.

The assessment team devised tests on the basis of the existing curricula. Apart from cognitive operations (knowledge, comprehension, application, problem-solving), these tests aimed to measure the degree of integration and transfer. Integration refers to essential information and extraneous information, whilst bringing into play knowledge acquired previously. The transfer objectives include behaviour learnt in a specific context being applied in a different context. The knowledge or know-how must therefore be used in different situations from the one in which they were learnt.

Taking place at the end of the school year, and based as it is on the curricula taught during the year, the assessment therefore aims to take stock. It is part of a summational assessment.

In the case of Djibouti, the curricula used as a basis for the assessment are directly modelled on French curricula. There are no plans in the short and medium term to change this state of affairs. At most, minor adjustments may be devised to attenuate the strongly negative impact of teaching applied from the start in a language other than the mother tongue. The results expected at the end of the primary and secondary education must be in line with the French curricula. The baccalaureate (school leaving certificate) awarded in Djibouti is automatically equivalent to the French baccalaureate, which enables some of the 200 successful candidates each year to go on directly to French universities.

Findings in primary school

At the end of primary education, an estimated 35% of the total number of pupils fail totally, with a distribution which varies from one part of the country to another according to the socio-economic level of the pupils. One wonders whether they can even read and write.

There are several theories to account for this state of affairs:

- the curriculum is too ambitious and does not correspond to the socio-cultural situation of the population, whose mother tongue is not the language of education;

- repeating a year is only allowed for 10% of the total class, which means that pupils have to move up into the next class when they are not equipped to continue their education to any advantage: the gap widens from year to year and the backlog eventually becomes irrecoverable;

- teacher training leaves much to be desired; in particular, there are too many supply teachers, without qualifications, at key moments in the early stages of schooling;

- pupils develop at their own rate, and are expected to progress far too rapidly for many of them; it is nevertheless hard to imagine a school with different curricula or varying educational methods, as in developed countries, as these prove to be very costly.

Some existing principles are nevertheless worth respecting:

- no pupils should be turned away during primary schooling;

- pupils should not be allowed to exceed the age limit when they could move on to the next stage in their education;

- class numbers should be kept in balance, at between 40 and 50 pupils for classes in towns.

Any solutions considered for remedying the situation should neither disrupt the present teaching structure nor entail an increase in the financial burden, in any form whatsoever.

There seems to be a critical moment beyond which pupils at risk are no longer able to benefit from education. According to on-site observation, this time comes at the end of the third year of primary school, when pupils are generally considered to have acquired basic reading and writing skills, as is assumed in the curricula for subsequent years.

It would be possible to set up a structure to allow systematic caching up for one year between the third and fourth years of primary school. Pupils who are behind at the end of the third year to such an extent that they obviously cannot follow the lessons in the fourth year normally should be detected at that time, then assigned for one year to a teacher whose main task would to make sure the pupils catch up linguistically. After a year catching up in a normal-sized class, the pupils would rejoin the primary school at fourth-year level. This would be a kind of programmed repetition, with the intervention of appropriate teaching methods, the choice of specially trained teachers and special monitoring from the point of view of educational support.

The competitive examination for entry into secondary education has peculiar effects which produce appropriate responses from teachers of the last year of primary school. The question is whether the competitive examination is relevant and especially whether it meets the objectives pursued by primary education.

The fact that the same types of questions appear as a matter of course each year encourages teachers to 'cram' the concepts which crop up regularly, to the detriment of studying the curricula. They also reserve all their attention to the pupils who seem to them from the start of the year to have a good chance of passing the examination. Training and improvement in original thinking are particularly neglected. Pupils fail the tests as soon as they are confronted with situations which cannot be reproduced.

To remedy this situation, the competitive examination needs to be reviewed to make it less amenable to cramming focusing on repetitive activities of little formative value and to open it up to situations which cover the essential concepts. The range of questions should cover all the objectives of primary education.

Findings in the first stage of secondary education

Results appear on the whole to be satisfactory in French and mathematics. They show the effectiveness of the teaching and the conscientiousness with which teachers perform their tasks. In particular, analysis of the different results shows the great consistency of the system in general and of the BEPC examination in particular.

A study of the results, especially by analysing the correlations and through factor analysis, highlights a certain number of problems to which it would be desirable to find solutions.

The correlation between BEPC oral examinations and the indicators as a whole is slight, which points to their lack of reliability. They should be abolished, especially as there are no rules for conducting their component tests.

The fact that the BEPC diploma is awarded to pupils obtaining very poor results reduces the general consistency. At present the cut-off point is too low. Indeed, the correlation is invariably weaker between passing the BEPC (decision) and the other indicators (report, selection and guidance, and tests). The diploma is devalued as a result. It would be appropriate to raise the BEPC, pass mark considerably to make it into a real examination.

Pupils lack motivation at the end of the fourth year, which can be seen from the behaviour of many of them at the tests. This is doubtless largely attributable to the low status of the BEPC, since selection and guidance for the fifth year is determined by the marks in the report for the whole year. Even though this selection and guidance process is actually effective (it is the best pupils who are guided towards grammar school), it would be necessary to give a certain status to the BEPC. The results should be used, at least in part, for selection and guidance for the fifth year.

The marks for French comprehension, already unsatisfactory in the third year, showed little improvement in the fourth. These comprehension problems are very probably caused by vocabulary difficulties, since French is not the pupils' mother tongue.

The shortcomings of the pupils in handling language as a tool mean that the teachers pay too much attention to technical questions (too much grammar), at the expense of comprehension and writing. Teachers adapt to the exercises on offer, in spite of this test forming part of the BEPC.

There is a break in French be tween primary and secondary school. For example, at the end of primary education, the pupils use the textbook for the fifth year for reading and grammar.

The assessment has made it possible to bring out the strengths and shortcomings of Djibouti's educational system. It is now up to the political authorities to shoulder their responsibilities and implement the proposals, which will make for improvements to the system, and to make sure that the existing achievements of education are safeguarded.

B.L.