7. Results of the EMP Testing in Primary Vocational Education Institutions. Workshop Summaries and Recommendations

The self-evaluation parameters suggested in the Project for primary vocational education institutions were piloted in 6 vocational schools and lycea in Almaty City and Almaty oblast including vocational lyceum N° 10, 15 in Almaty City and vocational schools N°8 (Almaty City), 11 (Kaskelen) and N° (Uzun-Agash).

The sample piloting was built on combination of training and analytical and review workshops with the pedagogical practices in the mentioned primary vocational education and training institutions regarding self-evaluation of their performance and following-up discussions and dissemination of the Project outcomes at the final round-table meeting. In total there were 4 major events and a number of the Project working group meetings arranged.

The first inception methodological workshop took place on December 20, 2001, attended by managers and deputy managers of the above mentioned institutions, the Project experts and representatives from the Republican Post-Graduate Teachers Training Institute. Participants of the workshop were presented the training and methodological package for self-evaluation in vocational education and training institutions, with the techniques of working with it explained. The package presentation was followed by a number of comments, proposals and reservations with the main conclusions as follows.

Vocational schools and colleges obviously need modern instruments for cost-effective and regular receipt of various information on their performance in the new social and economic environment. (All the managers present at the meeting expressed their interest in self-evaluation against the suggested parameters.)

Provisions of sections 1 and 2 related to status and forms of ownership of vocational training institutions, planning, staffing, physical base, funding, study plans and curricula, student’s record-keeping, attendance rate.

The suggested parameters for self-evaluation in vocational education and training institutions do not contradict, but supplement outcomes of external attestation of given educational institutions by central and regional education authorities.

The Project experts were recommended to pay attention to reasonability of points-base evaluation in the 1 section of the Package.

The pilot self-evaluation should be described precisely according to the procedure provided in the Package.

The initial piloting should take one month minimum.

Thus the main outcome of the first workshop was a number of amendments to the Package content and readiness of the management to piloting.

The second workshop “The Self-Evaluation Methodology for Primary Vocational Education Institutions: Outcomes of Piloting” was conducted more then a month later. The objective of this workshop was to discuss the experience of vocational lyceum in piloting self-evaluation based on the methodology suggested by the Project experts. The agenda for the workshop included presentation (by managers of these education institutions) of the outcomes of piloting self-evaluation followed by discussion, analysis and consolidation of successes and failures of introduction of the suggested methodology. The expected outcomes were determined by proposals to further elaborate self-evaluation indicators and procedures. The composition of participants had been expanded involving the City Education Department.

The following fragments of the participants’ presentations and discussions are considered by the Project experts as the most important for further improvement of the self-evaluation methodology.

VS #11 (Kaskelen): “It’s a relevant initiative. In the last year we were attested and it was very difficult without prior self-evaluation. While piloting the suggested methodology we established self-evaluation teams of teachers of general and specialized subjects, training officers and students. The teams’ activity allowed us to identify the weal point, the majority of them being in the field of management, to be more precise - in control and information support to decision-making. Higher specialized questionnaires (such as student’s evaluation of teachers’ performance, etc.) would be very helpful.”

VL N°15 (Almaty city): “It’s a lot of work and not yet completed. But even now it’s evident already that our main difficulties are associated with management in general and control in particular.”

VL N°1 (Almaty city): “The major problems are judged by every educational institution form its own position. It’s difficult to identify them precisely. We need models to compare with and on the basis of this comparison we can judge objectively our performance. Otherwise there will be cases of over-estimation or under-estimation. Outcomes of external attestation performed by education authorities can be used as an additional reference point.”

“The suggested self-evaluation can be further deepened by conducting at individual structural units and groups of students. In any event, the management style should clearly be changed. This is the main unavoidable conclusion after objective evaluation of our performance.”

VS N°18 (Almaty city): “2 years ago we were attested, the Regulations on Attestation proving for prior self-attestation. We tried to arrange that and emphasiÿed availability and adequate presentation of management-related and training documentation. The first reaction of the staff was a negative one, but in two weeks the majority acknowledged the reason and benefits of such evaluation minimum twice a year (every six month). The main advantage of the suggested self-evaluation methodology is that it is systemic and process- and result-oriented. During piloting we discovered our underused capacities.”

VL N°10 (Almaty city): “Currently self-evaluation is necessary. If you want to deliver really good performance you should control yourself constantly since it’s difficult to develop without knowing where your mistakes are. Outcomes of piloting the suggested methodology suggest the insufficient coverage of inter-action with parents of our students.”

Almaty City Education Department: “External evaluation indicators aim to profile the existing situation in the vocational education. But we need information on perspectives and reserves for development of these educational institutions. And this issue is addressed by the self-evaluation methodology suggested by the NRC. The given document embraces all the facets of operations of vocational education institutions. It provides ground to defend your positions against diverse inspectors form education authorities. Another advantage of the document is that it prepares the educational institution for external attestations. Proposal: to cover issues of early professionalization, development of poly-functional vocational training, bidding for training of unemployed, social partnership, methodological work.”

When reviewed and summarized, this and other opinions on the outcomes of piloting the self-evaluation methodology witness not only the need in such a document among management of primary vocational education and training institutions, but also their active and constructive position concerning finalization of content of questionnaires. The key outcomes of piloting of the methodology have proved that the most problematic area for vocational schools and lyceum is upgrading of the management process, structure and functions towards democratization, efficiency and adjustment to the market environment. The main threats for development are under-funding, high staff turnover, lack of reliable labor market needs assessment mechanisms, unstable economic development of certain territories.

To ensure broader dissemination of the outcome of piloting the round-table meeting was arranged attended by over 20 senior officials from oblast and rayon education authorities and vocational education institutions. This even was conducted in the form of a section in a course of the Republican workshop in Shimkent, January 25 of the current year discussing the state standards for primary vocational education and training. The agenda of the round-table meeting included the following issues:

· Goals and general scope of the UNESCO “Quality Control in Vocational Education and Training: Self-Evaluation Approaches” Project;

· Specifics of the methodology and organization of self-evaluation in vocational education and training institutions elaborated by the Project experts;

· Outcomes of piloting the self-evaluation methodology in vocational schools and lyceums of Almaty city and Almaty oblast;

· Capacity and perspectives of applying the Training and Methodological package for self-evaluation in primary vocational education and training institutions.

Discussions on 1 and 2 points involved presentation of summarized information on the Project, point 3 involved presentations made by heads of educational institution where the piloting had taken place. Point 4 became the main subject for discussion and argument. Having in general approved the suggested self-evaluation methodology, some of the participants of the round-table meeting expressed concerns on a possible conflict between self-evaluation parameters and outcomes of external attestation performed by the Ministry of Education and Science. Since the discussion proved the absence of any conflict or contradictions, the majority of participants recommended the discussed document for internal use in practices of primary vocational education and training institutions. The major comments related to funding and education (personal development) work. Certain managers were concerned with extreme - to their mind - transparency of financial performance of schools (lyceums), others suggested to include a block of issues reflecting personal development aspects of activity of vocational education and training institutions. Proposals of this kind witness persistence of traditional conservative approaches to management and evaluation of vocational education and training institutions still pursued by some of the managers.

We believe that the main result of the round-table discussion is awareness of oblast level primary vocational education and training management of the outcomes of the Project study and their interest in introduction of the self-evaluation methodology recommended by the Project. Another important result was amending questions in the questionnaire related to vocational education and training institutions’ budget.

During January 30-31 the final workshop was arranged to disseminate the outcomes of the Project, train new group of vocational education and training institutions managers in self-evaluation techniques and to arrange a round-table meeting to discuss further dissemination and introduction of the suggested methodology. The workshop was attended by about 50 participants including representatives form the vocational education and training system form 10 oblasts of Kazakhstan (oblast and city Education Departments, vocational schools and lyceums), Professional Training Institute of the Kazakh Academy of Education, non-governmental, international and other organizations. Similarly to the previous workshop, the audience was offered presentations made by Project experts followed by heads of vocational schools and lyceums where the self-evaluation methodology had been piloted. The agenda for the workshop envisaged involvement of previously trained heads of VS and VL as facilitators for the small group discussions. The groups were suggested to:

· Get acquainted with all sections of the self-evaluation questionnaire;

· To clarify the unclear issues with the support of the group facilitator;

· To test filling the questionnaire in - i.e. to answer the questions not requiring any additional information (individually);

· To discuss with other members of the group the difficulties encountered during filing in the questionnaire and propose further amendments;

· To perform the general SWOT analysis of the outcomes of the pilot self-evaluation;

· To fill in the “General Outcomes of the Pilot Self-Evaluation” table attached to the questionnaire (individually);

· To prepare suggestions for presentation of the results of the group work and the final round-table meeting.

The groups’ presentations suggest the following summary. Based on the pilot self-evaluation the most often reported strong sides of VS and VL are:

· Content of curricula;
· Availability of teaching engineers;
· Traditions to plan the work;
· Organization of training.

The following problems were reported as weak features:

· Physical training base;
· Budget;
· Jobs for graduates;
· Low management efficiency;
· External ties, social partnership;
· Poor learning achievements of students.

The following opportunities were reported by managers as underused:

· Participation in contests, national and international education projects;
· Decision-making techniques and procedures;
· Personal development and methodological work;
· Sponsorship;
· Advanced educational technologies;
· Professional upgrading of teaching and engineers staff;
· Students’ motivation;
· Meeting employers’ needs;
· Professional orientation;
· Self-evaluation;
· Earnings form the students’ production activity.

The majority of participants saw as the main threats:

· Decreasing Government order for workers training;
· Lack of demand for graduates at the labor market;
· Increasing competition at the education services market;
· Lack of affordable and cost-effective labor-market monitoring mechanisms;
· High staff turnover;
· Lack of necessary conditions for practical training of students.

The given list of factors shows that the bulk of answers was given in the section “underused opportunities” though in the course of all the discussions heads of vocational schools and lyceums were mainly talking about numerous problems mainly associated with budgeting and physical base and logistics. Thus, the conclusion may be that even during a very short period of time (the small groups were given 3 hours to work) self-evaluation according to the suggested pattern allowed to identify not just the problem zones, but considerable internal and external resources for the development of educational institution as well. Interestingly, the mentioned section covered factors reflecting advanced techniques for adjusting the vocational education and training institutions to the new social and economic environment.

The round-table meeting was offered the following questions for discussion:

· What and how needs to be done to incorporate controls and analysis into the regular professional practices of vocational schools and lyceums at all levels of management:

- School administration;
- Teachers and engineers staff;
- Students.

· How to arrange self-evaluation avoiding shocks and conflicts in the school team?

· What preliminary psychological and methodological preparation is required to ensure objective and comprehensive self-evaluation?

· How to motivate teachers and students for continuous self-evaluation?

· How to continue with dissemination and introduction of the recommended self-evaluation methodology?

During discussing these issues a lot of interesting and helpful ideas and proposals were presented including:

· To get all the staff and students involved, to introduce and develop the self-evaluation culture;

· To design additional set of questionnaires to evaluate more specific elements of the management and training structure (e.g. an individual lesson, practical classes, management decision, etc.);

· To systematize the detected negative factors and to highlight the most important ones;

· To train managers and teachers in analytical and evaluation results processing techniques;

· The optimal time period for educational institution - 2 weeks maximum (from the experience of pilot school and lyceums);

· To design analytical mechanisms to monitor job-placement and professional adaptation of graduates;

· To shift to a new management paradigm as a main precondition for adjustment to the new social land economic situation.

Based on the proposals of the participants of the workshop the experts made following clarifications and additions during finalizing the training and Methodological Package:

· 12 questions were more precisely worded;

· Questionnaires to evaluate classes and students’ evaluation of the classes were added too the Package;

· Duplicating questions were deleted;

· Recommendations were given concerning summarization of the self-evaluation outcomes;

· Preparation and actual self-evaluation procedures were given more detailed description.

Thus, the given workshop proved the meaningfulness, feasibility and efficiency of cascade training in self-evaluation methodology with trained experts acting as trainers for other managers. Another prove of the efficiency is that every meeting contributed to improving the content of the Project study.