|APPEAL - Training Materials for Continuing Education Personnel (ATLP-CE) - Volume 2: Post-Literacy Programmes (APEID - UNESCO, 1993, 112 p.)|
A. ROLE OF EVALUATION
Evaluation and feedback are most essential and important aspects of the administration, supervision, material design and conduct of any educational programme. Since the approaches advocated in this volume for the development of post-literacy are relatively new and untried systematic evaluation of all aspects is critically important.
Evaluation is an integral part of the programme. Its main function is to help arrive at educational decisions for the improvement of the post-literacy programme and the attainment of its objectives. Evaluation also serves to determine if materials need to be changed or modified to serve the purpose for which they were created. The materials are not to be judged as separate or isolated items, but should be assessed in relation to the objectives of a particular educational programme.
B. STAGES OF EVALUATION
In an evaluation, comparisons are made between the planning and the implementation of a programme. If the planning matches the programme implementation, then the programme is achieving its goals. To ensure a high correlation between planning and programme implementation, three stages of evaluation are needed:
1. Stage one: Pre-implementation evaluation;
2. Stage two: On-going evaluation, also called formative evaluation
3. Stage three: Evaluation at the end of implementation, also called summative evaluation
The relationships between these three stages and what is to be evaluated at each stage are summarized in Figure 9.1.
The diagram given in figure 9.1 shows that evaluation is a continual ongoing process.
Even after initial implementation monitoring must occur to detect aspects which could be improved and to determine procedures for improvement.
An aspect of special significance for Post-Literacy Promotion Programmes is the need to investigate long-term impact on the development of learning autonomy and the emergence of autonomous personalities in the society. Some measures should be attempted for assessing the contributions being made by post-literacy programmes towards the development of a learning society.
Figure 9.1: Stages in evaluating an educational programmes such as post-literacy
In post-literacy it is important that the above model be applied at all three levels of management - at level A (national), level B (provincial or regional) and level C (local). In particular the nature and suitability of the curriculum design and especially of the competency levels and their indicators of standards should be carefully appraised.
C. A CHECKLIST FOR EVALUATING A POST-LITERACY PROGRAMME
The following checklist sets out the main issues to be assessed during the evaluation of a post-literacy programme.
a) Background Information
In any evaluation, we need to know whether the planning correlates with the actual implementation of the programme. Therefore, a set of criteria is needed for specific aspects of the programme. These could include:
i) Budgeting aspects
Where does the money come from?
Is the programme for post-literacy accountable?
Is the money spent according to the plan?
What types of expenses are involved and how much money is needed for each type of expense?
ii) Economic effects
What economic sectors might benefit from the post-literacy programme?
Is the post-literacy programme likely to have any impact on the economy in general? If so, what?
Will it help break the vicious circle of poverty and contribute to the emergence of a spiral of prosperity?
How many target audiences will the post-literacy programme reach?
What are the development needs of these adults in terms of post-literacy materials?
What improvements are expected in language, numeracy and relevant mental skill at different levels?
Is the programme accepted by other agencies?
Do other development agencies recognise its importance?
Is the post-literacy programme likely to contribute to development of the nation? How?
Is the post-literacy programme likely to contribute to the development of democracy in the nation? How?
Will the post-literacy programme have a real impact on the society?
Will it develop autonomous learners and contribute to the emergence of a learning society?
Will the gap between the haves and the have-nots decrease as a result of the post-literacy programme?
vi) Technical aspects of the management plan of the programme
Are there sufficient resources?
Is the development process likely to be effective?
Are the expected outcomes realistic? Will they live-up to expectations?
Is there any built-in supervision of quality control measures?
vii) Administration plan
Are there any loopholes in the administration of the programme?
Will the lines of administration through levels A, B. and C function properly?
How well is the implementation plan organized?
b) Selection of Data
The information available to evaluators is often disorganised and confusing. However, the following types of information should be compiled for any evaluation:
i) Description and details of the organization of the project;
ii) Which types of methods should be used to check on the criteria to be used for the evaluation. For example, one criterion may require observation, while another could require a survey or an interview:
iii) A clear and precise statement of the kinds of data that are needed, the sources that will be used and the data indicators.
In the case of post-literacy programmes this needs to be approached very systematically because of the wide variety of possible presenters (providers) and the range of possible delivery systems.
c) Collection of Data
Collection of the data that will be used in an evaluation must be carried out in an organized fashion. The process of collection is indicated in the following five steps:
i) Step one - Develop evaluation instruments:
ii) Step two - Test the evaluation instruments;
iii) Step three - Organize appropriate samplings;
iv) Step four - Gather the data to be used by means of the following methods:
· Direct observation, if the information needed has something to do with administration, behaviour or the environment;
· Individual interviews, if the information needed has something to do with politics or other social issues;
· Previews of cumulative records of achievement;
v) Step five - Double-check the material gathered to determine whether the amount and quality of the data are adequate. If not, additional data gathering may be required.
D. MATERIALS EVALUATION
All types and formats of materials used for post-literacy require evaluation both before and after publication. Both pre- and post- publication evaluation should examine (i) technical aspects of the materials and (ii) reactions of the learners.
a) Evaluation of Reading Materials
The following aspects should be evaluated.
i) The Text
Simple words are to be used. They must be within the span of the vocabulary of the client group, except when it is necessary to intro duce new or technical words. In such cases, the new words need to be explained.
The sentences should fit each post-literacy level. (competency level a. level b. level c).
The reading material must be broken into short paragraphs. Use of headings and subheadings will help neo-literates better comprehend the content.
There should be enough space in the margin and between the words lines. Density of reading matter makes it difficult to read and understand.
5. Type face:
Printing type should be larger than the one used for the general public especially at competency levels(a) and (b). Ornate types should be avoided.
ii) Visual Elements (illustration, drawing, etc.)
The following criteria should be applied in judging the suitability and effectiveness of the illustration
1. They should be simple but attractive.
2. They must be relevant to the subject matter.
3. They must faithfully reflect the local situation - appearance, dress, houses, implements, etc.
4. There should not be too many details.
5. Use of too many colours is distractedly and costly.
iii) Arrangement of the Content
The following checklist is recommended for evaluating the presentation of content in reading materials for neo-literates.
1. The style should be pleasant.
2. The writing should not be like a sermon. It should be friendly and participatory, and not a string of instructions from a so-called knowledgeable person to a so-called ignorant person.
3. If possible, it should feature some human interest, e.g. a typical family facing a few problems. But care should be taken to see that the message is not obscured by the story.
4. Too many messages in a single book or booklet should be avoided, especially at level (a) of the curriculum.
5. The information given must be technically accurate.
6. The content matter must be relevant to the life of the client group and must try to answer some real need.
7. It must be within the capacity of the target group to try out what is suggested in the book or booklet.
iv) Reactions of the learners
Technical evaluation as described above can determine whether the book/booklet conforms to some of the essential norms which are necessary for such materials. It is a kind of technical evaluation which experts in the field of development and production of neo-literate materials can undertake. Additionally books also need to be tested by the learners for both readability and acceptability. Technical evaluation alone does not measure the suitability of the material. For this, reaction of learners can be investigated through (l)pre-testing drafts either in the form of photocopies or mimeograph and (2) post-production evaluation.
The materials have to be tested in three kinds of situations - (a) structured, (b) semi-structured and (c) unstructured.
· Structured situations are found in organized adult literacy programmes or continuing education centres where the graded materials are used.
· Semi-structured situations like libraries, mobile book vans or cycles and reading centres can yield valuable data, provided there is a conscious and organized effort through trained interviewers who are development workers and post-literacy instructors.
· Unstructured situations in the community itself. The materials have to be tested through community education programmes and by interested individual learners.
b) Evaluation of other learning materials
Evaluation of all types and formats of learning materials is necessary. Some criteria for the evaluation of the various types of audio-visual resources have been suggested in Chapter 4 and are not reviewed here. There are, However, several issues common to all types of learning resources to be used within a programme of post-literacy. These include the following:
i) Are the materials suitable for the environment and socio-economic back ground of the target group?
ii) Do the materials «fit» the curriculum? That is can they be located within one or more of the cells of the curriculum grid in terms of content and standards of competency?
iii) Are the inputs needed for production or dissemination available?
iv) Are the messages likely to be effective in developing learning autonomy, in helping to foster the emergence of an autonomous personality and in promoting a learning society?
E. ANALYSIS AND INTERPRETATION OF DATA
Once data are compiled, the evaluator must analyze the findings and present these along with recommendations for programmes improvements in a written format to programme planners and other personnel involved in the post-literacy programme. At this stage, the data should be explained in an easy practical and concrete style.
F. CONCLUSION: POST-LITERACY AND HUMAN RESOURCE DEVELOPMENT
This chapter has provided guidelines for evaluating a programme of post-literacy. In conclusion, however, it should be stressed that the procedures outlined above can be applied in both short-term and long-term studies.
Evaluation is important in this area not only because the approaches advocated in this volume are relatively new and untried, but because the impact of systematically applied programmes of post-literacy has been under-researched and under-valued. What is needed, perhaps, are systematic long term research studies on the various factors, including post-literacy programmes, which are promoting the evolution of a learning society. Such studies would be helped if there was in place an effective nation-wide Management Information System (MIS) for Continuing Education as a whole. Such MIS should first assemble appropriate base-line data against which changes and the impacts of the changes could be assessed.
Post-literacy programmes as part of Continuing Education make a direct and significant contribution to human resource development. Evaluative impact studies should check on the extent to which this has been achieved. The arguments in support of this proposition are outlined below, but there arguments need back up by long term evaluative studies on the impact of post-literacy programmes on social and economic growth.
The human population of a country can either be its asset or its liability. It is an asset if it is productive and possessing high work ethics. On the other hand it will be a liability if a greater proportion is unproductive, illiterate and unresponsive to the changing environment of the world. The society which is unresponsive and unproductive will always remain poor even though it is endowed with rich natural resources. It is often said that a person who is poor will be condemned to proverty for the rest of his or her life. The attitude towards life being negative such a person becomes a burden to the society. He or she is not a part of the labour force. Income is low while consumption is high. This trend sets up a vicious circle of poverty.
Is there any hope for a poor person to make a decent living? There is. If he or she is mentally strong this chain of poverty can be broken. That is possible if attitudes are positive. If poverty is the function of a poor attitude of the mind then positive thinking can produce reverse results. (See ATLP-CE volume I.) The quality of the mind decides the place of an individual in society and indeed the place of a society in the world. This explains the importance of humankind in relation to other aspects of nature. The rich natural resources of the world will be meaningless and unavailable unless humankind can act wisely to develop a sustainable environment for all. Newly industrialised countries (NICs) have come to understand this. Even small countries not well endowed with natural resources have become NICs when they have given high priority to improving the quality of their human resources.
Countries such as Japan, Singapore and the Republic of Korea are clear examples of this trend. In such countries a large proportion of the national budget is allocated to education and training. Aspiring countries such as Malaysia and Thailand which are rapidly approaching the status of NICs now see human resource development through education and training to be a key strategy in socio-economic planning. The Second Outline perspective Plan I 991-2000 section VII Human Resource Development of Malaysia makes the following comment (p. 25).
«During the decade of the nineties, human resource development will assume new importance. Competitiveness, productivity innovativeness and capability in management of new technologies in Malaysia will be determined by the quality of its human resources.»
The Malaysia report also makes clear that EDUCATION is the strategy for human resource development. The report adds:
«Human resource development must contain policies and programmes to continuously upgrade and improve the education and training programmes and facilities to meet the changing skill requirements,» (paragraph 1.87 page 25).
The implications of policies such as those mentioned above is that the policies will be maximally effective only if life-long education systems are developed. This means there must be a strong commitment to continuing education, and especially to the types of continuing education encompassed by post-literacy programmes. Whilst primary education is the cornerstone of formal schooling, post-literacy represents the «take off» point for all forms of continuing education. This is because post-literacy programmes promote autonomous learning and foster the development of truly autonomous personalities.
Post-literacy programmes, therefore, determine the quality and character of a societys labour force. Strong post-literacy programmes are the pre-requisites for an educated, learned and responsive labour-force for any community. Such a labour force ensures a good quality of life for all and permits each individual to grow to his or her maximum potential. See ATLP-CE Volume I.
In an attempt to guide Member States in the development of Post-Literacy Programmes within Continuing Education this volume has approached the issues of planning, designing and implementation systematically. Based on a systems model, it has set out a set of procedures as recommendations for consideration by policy makers and continuing educators.
The volume is intended as an exemplar and as a guide. Actual planning and implementation must take into consideration the political, economic and cultural more existing in any particular community. Needs and aspirations of a country as a whole must also blend with the needs of local groups. What the volume has tried to do, however, is to argue in favour of developing post-literacy programmes as key agents for socio-economic growth and hence for ensuring the improved well-being of all. It has also provided some frameworks and guidelines which may assist in fostering growth in this vital area of education. In evaluating the impact post-literacy programmes these aspects should be the central focus.