7.4 Determining the unanticipated outcomes and using these as benchmarks for future projects

Jorge Anguilar Rodriguez
The Autonomous University of Sinaloa

In this paper, Jorge Anguilar Rodriguez describes the method of assessment used in the Mexican Advanced Professionalisation Scheme (MAPS). He indicates that although the research design utilised in this project was similar to standard research designs used elsewhere, the emphasis in this kind of assessment is different. The emphasis in the research design was directed at uncovering inter alia the unanticipated outcomes – and these, once discerned, played a significant role in ensuring project sustainability. In addition, he indicates that, as a by-product, these outcomes contributed to the development of new projects. The positive unanticipated outcomes were posed as benchmarks for the continuation of the MAPS programme. The assessment also enabled researchers to discover capacity among teachers. Several teachers showed enthusiasm for as well the ability to train new cadres – and this contributed to sustaining MAPS.

1 Introducation

The Autonomous University of Sinaloa (hereafter referred to as UAS) is located in the state of Sinaloa in Northwest Mexico. It has a population of 95 000 students. 11 630 of those students study at the four Language Centres of UAS in Los Mochis, Guasave, Culiacan, and Mazatlan. There are two different programmes in the language centres, the regular course for young adults and adults (which has an enrolment of 8 480 students), and the Saturday Children's Programme, with an enrolment of 3 150 children with ages ranging from 8 to 14 years old.

Both the regular courses and the children's programmes offer general English courses that teach the kind of communicative competence that children need and that adults need to make them effective communicators in their personal, academic and professional lives.

Methodology

In order to ensure the success of the impact evaluation and to have a clear framework for our evaluation, we decided to conceptualise this investigation as a process consisting of the following steps:

1. Formulation of aims
2. Description of practice
3. Focus of investigation
4. Research instruments/data collection
5. Data analysis and interpretation
6. Conclusions, new goals, new projects and benchmarks
7. Dissemination of findings

2.1 Impact assessment as contributing to project sustainability

It is significant to note that although the above outline is similar to the conventional stages of all research enterprises, it was applied in such a way that the data gathered would be useful for the enhancement of project sustainability. Because the assessment needed to give as much attention to gauging the anticipated outcomes of the project as it did to gauging the unanticipated outcomes, each stage of the research design was considered for the what it would reveal about unanticipated outcomes. Thus, for example, stage 6 of the research outline, dwells on the importance of the establishment of new goals, new benchmarks and new projects. These arise from the uncovering of unanticipated outcomes in stage 5. For the same reason, stage 5 devotes a considerable amount of time to identifying what positive unanticipated benefits might be ascertained from project players. The subsequent phase refers to ways that such benefits could be mainstreamed so that new benchmarks might be formulated and new projects considered in stage 6.

2.2 Aims

We decided, prior to beginning the impact study, that we needed to formulate the aims of the investigation so that we could undertake more focused research. The aims specified at that time (April 1997) were:

· to investigate the extent to which individuals who have received training have contributed to the development of our institution

· to identify the expected and unexpected effects of the changes undertaken

· to identify changes in knowledge, skills and attitudes with reference to teaching, learning and language

· to become aware of our strengths and weaknesses

· to collect data that would provide us with findings which could be used to develop a plan for future development.

2.3 Description of practice and its effectiveness

After we had formulated the aims of our impact evaluation, we felt the need to describe our practice and programmes at the language centres prior to the implementation of the Mexican Advanced Professionalisation Scheme (MAPS). We did this so that we could familiarise ourselves with our teaching and management practices. We believed that such a description would lend weight to the evaluation and increase its validity and reliability. A team of teachers, administrators and academic co-ordinators were therefore subsequently involved in the process of describing practices and determining how effective they might be for meeting the needs of students, teachers, the institution and the community.

2.4 Focus of investigation

In order to be properly focused and avoid generalisations, we decided to determine, by way of analysis, the key areas that needed investigation. It was decided, after analysis, that the following areas needed assessment:

(1) Knowledge, skills and attitudes
(2) Teaching, learning and language
(3) Curriculum components

· syllabus
· materials
· assessment
· goals

In addition to these, it was necessary to gauge the expected and unexpected benefits as well as positive and negative effects of the project. This of course had implications for the choice of the research instruments.

2.5 Research instruments

Once we had agreed on the focus of investigation, we analysed various research instruments in order to find those that would be appropriate for assessing the areas to which we had assigned priority (those listed above). We found that practical and easy-to-implement instruments seemed to us to be the most appropriate. In order to make this part of the process more valid and reliable, certain contextual factors were taken into consideration. The research instruments which we ultimately chose were interviews, questionnaires, surveys, group activities and documentary evidence.

2.6 Data collection

We then decided to interview a few teachers on an individual basis in order to arrive at an understanding of personal involvement in institutional change. We planned, in this way, to collect data about individual and institutional change.

· Teachers were firstly asked to comment on their performance and on how they had viewed themselves before, during and after their training. To our surprise, we found that the unanticipated benefits and outcomes of the training that teachers received seemed directly related to the degree of their involvement.

· Secondly, we involved teachers in a group activity which comprised a series of tasks which would give us information about the effects of professional and institutional change. The teachers who were thus involved stated that they were surprised to discover how they had developed as professionals. They also indicated that they were willing to help in the development of our institution - thus contributing to project sustainability. The unanticipated benefits identified in the first part of this process were reinforced and discussed by the teachers in group activities. Questionnaires and surveys were used to collect the remainder of the data.

2.7 Data analysis and interpretation

Once the data had been collected, we met to analyse and interpret all the data. We were both surprised and gratified as we identified, at the meeting, more evidence of both anticipated and unanticipated outcomes and benefits. This process also made us more aware that we needed to create the conditions which would maximise the potential inherent in the training, attitudes and willingness of teachers to participate more actively in the development of new programmes and projects in our institution.

We may say, by way of summary, that the data analysis and interpretation stage made us aware of the hitherto unrealised potential of our situation, and this motivated us to embark upon new attempts to professionalise English language teaching. Such attempts were necessary if we hoped to improve the quality of the service given to the community - not only in our institution but also in other institutions, both public and private.

2.8 Analysis of anticipated and unanticipated benefits

The unanticipated benefits, which were discerned in the process, were:

· greater professionalism
· better problem management/identification
· better job opportunities for women
· having more women in key positions
· institutional development
· interest in teacher training and education
· interest in postgraduate education
· more academic dialogue
· decision making that results in learner benefits
· awareness of the teachers' role in the education system
· awareness of change
· a new conceptualisation of teaching, learning and language
· learning how to learn and autonomous learning
· more learner-centred decision making more reflective/analytical teachers
· an interest in research
· management of change

3 New goals, benchmarks and projects

The research process played an important role in identifying the benefits of both the anticipated and unanticipated outcomes. The determination of unanticipated outcomes in particular played a profoundly significant role in shaping how the programme would be conducted in the future. The positive unanticipated outcomes illuminated possible ways of taking the project forward and also suggested other, related possible projects. It was for this reason that the assessment focused heavily on the implications of the unanticipated outcomes in our context and institution as well as on what these outcomes meant for the project and for sustaining practice. The positive outcomes (both anticipated and anticipated) were treated as explicit benchmarks and goals for our current and future projects and programmes – and hence for the sustainability and further enhancement of the project.

As a result of the identification and definition of expected and unexpected benefits, we initiated several changes in our current programmes and projects and gave every encouragement to those who would have to carry them out. All these factors had far-reaching effects on project sustainability.

4 Dissemination of findings

We then attended a conference in Tijuana, Mexico, to present, describe and share the data which we had collected, the anticipated and unanticipated benefits, and our perceptions of the impact which the changes had effected. During this same conference, representatives from universities in north-west Mexico presented the impact which the Mexican Advanced Professionalisation Scheme has made in their institutions.

In addition, we organised a meeting in which we shared all details of our process as well as the expected and unexpected outcomes which we had identified and which are the subject of this paper.

5 Conclusion

We should like to state, in conclusion, that there are implicit and explicit benefits which arise out of impact evaluation studies. The most important of these, in my view, are the following:

· Impact studies raise awareness of the potentials and weaknesses of an institution.

· Impact studies make administrators and teachers aware of their new knowledge and skills, and their academic potential for developmental purposes.

· Teachers become aware of their important role in the education system

· Teachers become more involved and confident, and show a willingness to support the development of English language teaching.

We have become aware that English language skills are an essential asset for the development of the Universidad Autonoma de Sinaloa and for the whole of Mexico, and that our role as teachers and/or teacher-trainers is of paramount importance for the development of these skills. Finally, we should like to affirm that the benefits of the findings for project sustainability were immeasurable.