(introduction...)

It is recognized that school textbooks are one of the factors which determine the effectiveness of the teaching and learning process in school. They do, therefore, have a positive impact on the yield of educational systems. For this reason their quality is of the utmost importance so far as their relevance to the objectives and content of curricula is concerned and their value as a method of learning and acquiring knowledge.

Before the production of school books enters into the actual production phase, the final manuscript has to be evaluated to ensure that the text offers all the guarantees of quality and corresponds to learning needs.

Whilst it is being written, parts or successive chapters of the text are submitted to readers for assessment: This enables a partial evaluation of the manuscript to be made and permits the author to correct the text or make necessary revisions to improve its quality. However, only when the final manuscript is submitted can a complete and detailed evaluation be made.

Aspects of evaluation

Aspects which should be evaluated correspond to those defined as determining the quality of textbooks:

- Content
- Pedagogical approach
- Language
- Illustrations

All these aspects are equally important and their level should judged to be at least very satisfactory if the final manuscript is to be approved. Imperfections or an inferior level in one aspect will inevitably have negative repercussions on the others. For example, poor quality illustrations, which are blurred or inaccurate will distract the reader when he tries to relate them to explanations and information contained in the body of the text. Similarly, phrases which are too long and complex, or too many unfamiliar words and terms, can impede comprehension of the text and discourage the pupil's interest in content.

Presentation and composition of the text should also be satisfactory. The author should follow the technical guidelines recommended for presentation and composition and may also ask for guidance and assistance from the publisher.

These aspects of evaluation are, in general, all valid for textbook manuscripts. However, differences do exist, depending upon the discipline under consideration and the year or level of instruction. It is evident, for instance, that content, pedagogical approach, drafting and illustrations for a first year reading primer will not be the same as those for an final year junior school textbook for literary studies and prose. These same aspects will also differ for a mathematics, science or geography textbook. Before applying evaluation criteria, therefore, the textbook manuscript must be situated in relation to the aspects evaluated and its final destination. It can be situated in the following manner:

Content
	Discipline
Pedagogic approach
	Year or level of instruction
Drafting	Type of textbook for who

Chapters III and IV should be referred to for an analysis of the evaluation of these aspects.

Questions to be raised in the evaluation of a final manuscript

All the following suggested questions can serve as guidelines for evaluation. Some can be more or less important, depending upon the discipline under consideration, the teaching objectives and the level of instruction.

Content 1. Does content correspond to the objectives of the curricula for the school year under consideration? 2. Does the level of content correspond to what pupils have previously learned? (For the first year of schooling, previous results refer only to whether or not the pupil has had any pre-school education) 3. Is content accurate? Concise? Topical? Varied? Sufficiently objective? 4. Is it sufficiently enriched in relation to the content summarized in the curriculum? 5. Will it stimulate and develop positive social and moral attitudes? 6. Does progression in learning and concepts transmitted by content go from the simple to the complex, from the elementary to the advanced? 7. Are concepts clearly explained and does content offer practical applications? 8. Do learning activities and experiments proposed in the different chapters help to enhance learning? Are they varied? 9. Do such activities call for the intervention of the teacher? Or can they be carried out by the pupil without the teacher's help? 10. Has the author tried to integrate elements from two or more disciplines where possible and desirable? Educational Approaches 11. Does the method applied in the textbook correspond to a pedagogical concept recommended by the educational authorities? 12. Does the method correspond to a constrained, interventionist pedagogy? Or, on the contrary, is it inspired from open learning methods, inciting the pupil towards some autonomy in learning? 13. Does the manuscript propose research activities for the pupil: observation, surveys, measuring, information? 14. Are the interests of pupils taken into account to stimulate learning and has the author included motivational aspects (varied topics, familiar environment, evocative illustrations, etc)? 15. Has the author taken account of some of the pupils' characteristics, such as age, environment, to arouse their interest? 16. Are there learning exercises, and assessment for each chapter? 17. Are exercises varied? Well adapted to the content of the chapter? Do they seem too difficult or too easy? 18. Are exercises formulated clearly and concisely? 19. Can these exercises serve as a method of evaluating learning? 20. Are they prepared in such a way that the pupil can verify his own progress and success (for instance, answers contained in an appendix, solution routes proposed)? Drafting Organization 21. Does the manuscript clearly reflect the existence of logical organization of the text (Presentation. Main text divided into chapters and sub-chapters according to the curriculum. Summary. Exercises, Evaluation)? 22. Are some chapters organized in a different way from others, destroying the balance of the overall text, or is this difference justified? 23. Does the length of chapters correspond to the importance of themes? 24. Are some chapters too long, despite the importance of the topic? Or, too short? Language 25. Is the language used in the text easy to understand and adapted to the level of the pupils for whom the textbook is intended? 26. Does the vocabulary used correspond to the assumed level of the pupils, taking into account previous knowledge? 27. Is subject-specific vocabulary or terminology used? Are definitions given of new or unfamiliar words? 28. In general, is the length of phrases and their structure well adapted to the comprehension level of the pupils? 29. Is the meaning of the text clear and intelligible? 30. Is the punctuation in the text always justified? The continuous link or thread 31. Where content permits, has the author used a continuous link or main theme to arouse the interest of the reader and incite him to further study the text? Summary 32. Is the summary placed at the beginning or the end of the chapter? Is it clear, concise and does it correspond to the main aspects of the text? The style of writing 33. Is the style sober and efficient or are there too many enumerations, imprecisions, dramatisation? First and last pages 34. Does the introduction or the presentation clearly establish the objectives of the book and the meaning of the subject? 35. Is the table of contents detailed and accurate? 36. Were appendices and an index foreseen and have they been provided? Illustrations 37. Are all the illustrations contained in the textbook fully justified? Or, are there too many? 38. Do they correspond to the content of the chapters in which they appear? 39. Are they clear and accurate? 40. Do they transmit interesting information, and visual representations unfamiliar to pupils or outside their environment? 41. Are illustrations evocative and will they arouse the pupils' interest? 42. Has too much attention been paid to aesthetic quality of illustrations to the detriment of their role of transmitting accurate information? 43. Are they in the right place? Is their location clearly indicated in relation to the texts to which they refer? 44. Are accompanying titles and captions clear and accurate? Have they been numbered? 45. Is their cost justified. (For reproductions of photos, works of art, processing of original images, printing, number of copies of the book foreseen?)

Guidelines for evaluation

To standardise and facilitate the work of evaluators, an evaluation sheet can be used. For the-sake of commodity, it is preferable to use one sheet for each of the aspects evaluated. The following is a suggested lay-out:

Aspect evaluated:	CONTENT
Questions posed:	Replies			Comments and suggestions
	Yes	No	In part
1..........................
2..........................

The evaluators

Evaluators must fulfil certain conditions to participate in an evaluation:

- Possess excellent qualifications and professional experience in a given educational domain (teaching, curriculum, educational psychology, educational research, publishing school textbooks, elaboration of school books).

- Not be personally engaged in the elaboration of the manuscript under evaluation.

- Possess or have previously received full information on the different stages and operations of textbook production (seminars organized by a Textbook Committee...)

- Have received special preparation for the work of evaluation and know about all aspects of the elaboration of manuscripts (preparatory working sessions under the guidance of a competent expert who could be the Chairman of the Committee).

In principle, evaluators are members of the Textbook Committee. However, outside personalities meeting the requirements for participating in the evaluation can be called upon to collaborate in this work.

Division of work amongst the evaluators of aspects evaluated

Aspects	Evaluators
Content	- Subject specialists: teachers inspectors, supervisors
	- Secondary school teachers
	- Curriculum specialists
Pedagogical approaches	- Subject specialists
	- Educational psychologists
Drafting	- Language teachers
	- Publishing organizations
Illustrations	- Subject specialists
	- Graphic artists
Presentation and composition	- Publishing services (compositors)

Division of work is not mandatory, but simply indicative of the need to make use of all available qualifications. For countries having resources sufficient only for the production of one single textbook per discipline per school year, the evaluation of a manuscript can be the determining factor in obtaining the best possible quality of school books.

(introduction...)

Basically, the objectives of evaluation are to identify evident and important problems which appear in the manuscript and make recommendations to the author as to the corrections and revisions which will make his book a valuable teaching tool.

Evaluation is carried out by members of institutional groups (committees, or commissions) according to modalities which vary, but of which the main lines can be defined:

- a small group of evaluators is appointed to make a detailed reading of the manuscript. Reading is based on an guidelines which should assist all those participating in the evaluation;

- each evaluator carefully reads the text analysing in particular those aspects for which he is most qualified. If an evaluation sheet is used, he replies to questions and notes his comments, observations and judgements. In addition, he makes recommendations regarding any eventual corrections or revisions;

- the evaluation group meets and the evaluators compare the results of their analyses and recommendations. A discussion, monitored by the director or chairman of the Committee enables agreement to be reached on the conclusions of the manuscript's evaluation, for instance:

- very favourable, with only minor revisions which are then submitted to the author;

- favourable, with a certain amount of revision and improvement indicated by the Committee;

- less favourable, in which case the author is invited to redraft certain parts of the text.

- It is rare that a final manuscript is rejected, insofar as partial readings during its elaboration will have enabled judgements to be made on the capacity of the author to produce a satisfactory manuscript.

Evaluation by teachers

The final manuscript, evaluated by members of the committee and subsequently revised by the author, may then be submitted for evaluation by teachers, which is carried out under the supervision of one or two designated members of the committee. It can take the following form:

- successive detailed reading by several teachers (four or five) chosen according to the subject they teach, at what level and their qualifications. Each reader is accompanied by an evaluator who notes down the teacher's observations and comments;

- shortcomings observed and suggestions for corrections are discussed with the evaluator to reach decisions which will be submitted to the committee for approval.

Certain criteria should be applied to guarantee the validity of opinions expressed and modifications suggested:

- successive readings by each teacher should be made without knowledge of the observations of the previous teacher(s);

- imperfections observed and corrections suggested by only one teacher will not be taken into account if they are of only minimal importance;

- on the other hand, if three or four, or four out of five teachers indicate the same shortcomings, then these must be carefully examined and the suggested corrections submitted for the approval of the committee.

Another method can consist in submitting the manuscript to a small group of selected teachers and requesting them to note down any imperfections. Once the reading is complete, a meeting is organized for a group discussion under the guidance of one or two evaluators who take note of the participants' observations and comments. Suggested corrections are also noted and submitted to the Textbook Committee which will decide on action to be taken.

(introduction...)

Many countries do not yet have sufficient experience in the elaboration of textbooks and their generalized use at all levels of schooling. In such cases, a trial edition can avert serious drawbacks and waste of resources which would result from the mass production of school books which contain shortcomings which could be prejudicial to the quality of teaching. Such cases can occur in a country where there are no competent authors, or no teachers with sufficient experience in writing textbooks. For these reasons a trial edition should be tested, the main objective being to evaluate the effectiveness of the final, revised textbook in the teaching-learning process within a real school situation.

A trial edition can be produced with relatively limited resources, using, for instance, mimeographed reproduction or printing on small off-set presses. Illustrations playing an important role in the text should be reproduced, unless reproduction involves complicated techniques (such as the use of several colours). Reproduction in black and white can often suffice.

Copies may be limited to a few hundred.

Testing

The trial edition is produced once the final manuscript has been evaluated and revised. So, it is guaranteed, a priori, as relevant teaching material. However, testing in schools can reveal imperfections which have passed unnoticed during evaluation. The following conditions are applicable for testing a trial edition:

- The trial edition is tested in a small number of schools and in classes which correspond to the instructional level of the textbook (four or five schools);

- The schools chosen should be located in different zones (urban and rural or semi-rural);

- Teachers taking part should have good or average qualifications; if qualifications are too low, much of the validity of the testing and evaluation will be lost. Moreover, these teachers should be capable of expressing valid opinions on the experimental textbook and of making pertinent suggestions as to eventual modifications;

- The teachers designated to take part in the experiment should understand the objectives. They should also be aware that their observations and suggestions will be transmitted to the author(s) and to the publisher;

- An evaluator will accompany and supervise the experiment.

Evaluation

The reactions of teachers and pupils to the use of a trial textbook will serve as a basis for evaluation. After testing, the teachers should be able to answer the following questions:

- Is content interesting for the pupils?

- Is content well adapted to the objectives of the subject and the instructional level?

- Can the progression of content be followed without difficulty?

- Is the vocabulary of the textbook well adapted to the age and level of the pupils?

- Is the text sufficiently comprehensible for the pupils?

- Are illustrations clear and accurate enough and do they interest the children? (In this respect, the quality of reproduction of illustrations in a trial edition should be borne in mind).

- Is the pedagogical methodology which inspired the textbook easily applicable and is it adapted to the learning objectives and the level of instruction?

- Are exercises easy or difficult for the pupils and do they enable good evaluation of learning?

On the use of the textbook:

- Is the textbook easy to use and practical both for teacher and pupils?
- Do teaching conditions permit fully efficient use of the textbook?

Observations and suggestions from teachers

Once the teachers have completely understood the objectives of testing and with the help and advice of the evaluator, observations and suggestions can be made during actual teaching practice. These observations and suggestions, examined and recognized as pertinent by the group of evaluators can therefore be applied, if need be, to correct or immediately revise the manuscript without waiting for the end of the experiment.