Cover Image
close this bookEvaluating Impact - Education Research Paper No. 35 (DFID, 1999, 262 p.)
close this folder3. STAKEHOLDER PERSPECTIVES
View the document3.1 Identifying stakeholders
View the document3.2 Considering the audience - an important phase in project evaluations
View the document3.3 Impact studies and their audiences

3.1 Identifying stakeholders

Dermot F. Murphy
Thames Valley University
Pauline Rea-Dickins
University of Warwick


This paper focuses on how important it is for evaluators to identify stakeholder groupings if they want to make effective use of participatory evaluations in educational development projects. It argues that it is necessary to pay detailed attention both to identification of stakeholder groupings and to understanding their relationships to projects in question.

The authors provide a detailed exposition of the variety of ways in which the concept stakeholder may be defined. Moreover, they argue that these definitions generally pertain either to individuals/groups who are involved in or who are affected by a project or its evaluation, or to the differing interests of these individuals/groups. The authors argue that stakeholder interests are not usually rigorously defined. They indicate that stakeholder interests do not seem to offer the kind of insights that might guide the planning and management of participatory or stakeholder evaluation. This lacunae, they argue, suggests the need to identify more robust parameters for exploring stakeholder perspectives on evaluation.

Accordingly, the paper begins with the authors' undertaking an interrogation of the multiplicity of definitions of stakeholder. This is followed by their examination of the inherent power relations and power differences which, they indicate, provides a framework for exploring the nature of the roles and relationship of stakeholders in an evaluation. The authors then suggest three propositions about stakeholder perspectives, which they support by using data they have obtained from their research into the stakeholder problem. Their paper concludes by suggesting some of the implications that their findings might have for practice.


1 Introduction

It is essential for those who wish to make effective use of participatory evaluation in educational development projects to identify stakeholder groups and to understand their relationships with one another and to the project. There are various ways of defining the concept stakeholder and many of these refer to individuals or members of groups involved in or affected by a project or evaluation. These definitions usually centre on the differing interests which distinguish groups - an approach which, we argue, does not offer the information needed for guiding the planning and management of participatory or stakeholder evaluations.

Our experience of conducting evaluations, both as external evaluators and consultants in participatory evaluations, suggests the need to identify more robust parameters for exploring stakeholder perspectives on evaluation. In the next section we shall examine definitions of stakeholder before we look at power relations and before suggesting three propositions about stakeholder perspectives. In section 4 we shall explore stakeholder perspectives by looking at data from questionnaires, interviews and field notes and we shall try to establish the extent to which there is support for the proposed framework. In section 5 we will attempt to show the implications of our findings for practice.

2 Definitions of stakeholder

The notion of participatory evaluation in education is not a new one. Morris (1990: 131) indicates that in his original conception of evaluation, Tyler regarded evaluations as a tool to help the teacher in planning the curriculum and making instructional decisions. In the same place Norris adds that Tyler and Waples advocated the study of classroom problems by teachers and supervisors as early as 1930 - thus showing that both these authors believed in the usefulness of participatory evaluations nearly seventy years ago.

More recent discussions of stakeholder evaluation tend also to talk about the need to respond to the interests of real people and the irrelevance or even failure of other approaches to evaluation (Weiss 1986). The aim of a stakeholder evaluation is to make evaluations fairer and more useful, usually by getting primary stakeholders - the real people who may benefit from and/or implement the project - involved in conducting the evaluation of the project, in line with the proposal just cited.

These proposals are often criticised because some advocates of stakeholder evaluation blur distinctions between, for example, accountability and knowledge evaluation, and a few make the disputable claim that stakeholder evaluation is a sufficient approach to evaluation -with the implication that no other is needed (Chelimsky 1997: 22). It is unclear whether these proposals reduce privilege or pass it to different stakeholders in a participatory evaluation. In any case, this usage of the concept of stakeholder evaluation is misleading since all evaluations are conducted by or for stakeholders. The question is rather Which particular groups of stakeholders commission, use or do evaluation? It is recognised that no one seems to propose that all possible categories of stakeholders should participate.

2.1 Ways of identifying stakeholders

Stakeholders are frequently identified by their working role within a programme, or by their contribution to the programme. In such a case, the term stakeholder may refer to either individuals or groups. When stakeholders are defined by their working role within a programme, the definition is usually unclear about whether the definition is specifically to do with their place in the project or whether this classification refers only to their association with the evaluation.

For example, Rossi and Freeman (1993: 408; Weiss 1986: 151) refer to the following in their list of stakeholders:

Policy-makers and decision-makers... Program sponsors...
Evaluation sponsors... Target participants... Program
management... Program staff...
Evaluators... Program
competitors... Contextual
stakeholders... Evaluation
community....


This list is not exhaustive and identifies groups which, while they may not always be involved in carrying out the evaluation, are potential audiences for the findings. A similar categorisation by Aspinwall et al (1992: 84-85) tries to simplify the matter of classifying stakeholders by proposing four broad groupings:

Clients or customers

Those who are intended to benefit from the project

Suppliers

Those who implement or provide resources for the project

Competitors or collaborators

Usually other organisations

Regulators

Any agency which directly or indirectly regulates the project

The categorisation of Aspinwall et al has the advantage of not being an open-ended list, as is the one previously referred to. One could easily add to their list. We wish, however, to argue that the four categories of Aspinwall et al are not sufficiently distinct and consequently of little use. For example, some participants, such as teachers in an educational development project, fall into the categories of both client and supplier. There is, moreover little discussion about how the categorisation is arrived at, and it is not evident from such a list why and how each group will take a particular attitude or set of attitudes to an evaluation.

Rossi and Freeman (1993: 409) also focus on the multiplicity of stakeholder groupings. They point out that as a consequence of the multiplicity of stakeholder groups, evaluators may be unsure whose perspective they should take in designing an evaluation. This dilemma is interpreted by Hopkins (1989) as pointing to different groupings within the group of evaluators. He draws attention to the divided loyalties of evaluators who have to take the concerns of multiple stakeholders into account. They may (variously) be loyal to the:

Profession

Rossi and Freeman's evaluation community

Sponsor

Rossi and Freeman's sponsors

Community

Rossi and Freeman's target participants (The evaluator acts as advocate and these stakeholders are not actively involved.)

If one looks closely at each of the above classifications, it is immediately apparent that one could go on subdividing each of the groupings - since even stakeholders may also have divided loyalties.

The following classification, elucidated by Guba and Lincoln (1989: 40-41), takes the relationship between any stakeholder and the evaluation as the defining parameter. They then identify the following three broad groupings:

Agents

those who conduct and use the evaluation

Beneficiaries

those who gain from use of the evaluation

Victims

those who are negatively affected by the evaluation

As with each of the aforementioned classifications, various subcategories within each of the three main classes may also identified. This clearly locates each member's or group's stake as being part of the evaluation whereas the other categorisations were potentially indeterminate between their stake in the project and their stake in the evaluation. Again, however, when we apply the categories to familiar cases, some members seem to fall into two categories. There is a further difficulty that Guba and Lincoln (1989: 202) acknowledge, the difficulty of identifying victims. We suggest that it would also be difficult to predict which of the second and third categories stakeholders would fall into: our goal for a more comprehensive framework requires some predictive power.

It is common, then, to acknowledge that there are different categories of stakeholder, and that each category has its own interests and spheres of action. This notion, however, remains on the level of generality and is a taxonomy. We suggest that, just as Linnaeen taxonomies are revealed by plant genetics to misclassify species, a study of underlying factors in groups may reveal more about their workings. At this juncture, we feel that the most useful pointer is to recognise that the defining interest is the stake in the evaluation. Categorisation as a defining procedure is more or less observable, but this has little or no explanatory value when one tries to account for different stakeholder perspectives. We suggest that a more effective explanatory procedure is still needed.

3 Defining stakeholder perspectives

As we have shown in the previous sections, stakeholders may be classified as belonging to different groups, a small, select number of which, in the field of evaluation, have traditionally been involved in conducting evaluations. In order to extend involvement in evaluation (and thereby incidentally expanding the kinds of evaluation that may be undertaken), it might be useful to expand on the stakeholders' understanding of the evaluation process. The following is a comment by a stakeholder (practitioner) who is not traditionally involved in project evaluation (except as a more or less willing subject)1. His perception is that 'evaluations are done for the funding body by ex-patriate visitors'.

This view resonates what Rossi and Freeman (1993: 252) term a connoisseur evaluation, i.e. an evaluation done by an outsider who is a subject specialist not trained in evaluation - what they call 'among the shakiest of all impact assessment techniques'. In this case, a practitioner would be looking for power, the power to do and be involved in evaluation of his/her project.

3.1 Power as a variable in defining stakeholders

Power as an element of evaluation and the activity surrounding evaluation seems to have aroused curiously little attention if one accepts that evaluation and its utilisation are about the exercise of power: evaluations decide about the continuation or curtailment of a project and about the future direction of a project. We draw on the following definition of power.

Power is the ability of individuals, or the members of a group, to achieve aims or further the interests they hold... How much power an individual or group is able to achieve governs how far they are able to put their wishes into practice at the expense of those of others (Giddens 1989: 729).

Following Giddens, power (a basic sociological concept) refers to relations within and between groups, between individuals. We would add that, when applied to the process of evaluation, power is structurally created and allocated.

3.2 Knowledge as a variable in Defining stakeholder

It is pertinent at this point to draw attention to the relationship between knowledge and power. Power is dependent on knowledge. In some views, the level of this dependence, for certain modalities of power, is currently greater than ever before (Fairclough 1989). Evaluation, on the other hand, is about generating knowledge, whether general or specific, and, as such, has its own power. This power is greatest where the findings or knowledge derived from evaluation offer clear guidance to specific stakeholders about future action or where information for prediction is information for control-not forgetting, however, that some information/knowledge is not useful or is rejected by those in power (Patton 1997: 348-350). The relationship between knowledge and power suggests why those who hold power may tend to resist the process of evaluation activities: change in the conduct of the evaluation may lead to the restructuring of power within or between organisations.

If the exercise of power is about furthering group or individual interests, and stakeholder groups can be defined by interest, then it appears to be worthwhile to explore interest and power relations as parameters if one wishes to understand stakeholder relations and perspectives. This definition of power makes it clear that power may be relative and may depend on the ability of stakeholders to control the actions of others - regardless of whether the ability to control is ascribed or achieved. It is this experience of power that may also underlie the sense of disempowerment evident in the rejection of outsider evaluation cited above.

3.3 Knowledge, power and interest as variables in defining stakeholders

The exploratory nature of this work will become evident in the discussion that follows, particularly as it becomes increasingly evident that both interest and power are perhaps more complex than might prima facie appear to be the case. In an early stage of our empirical investigation into the notion that an understanding of power relations might illuminate stakeholder perspectives in evaluation, we considered a number of potential areas where relative power and different interests might come into play. These are:

Knowledge

About the project and about project evaluation

Expertise

Relevant to the project and to evaluation

Control

Power to initiate or stop action and participation

Budget control

Power to take decisions about spending

Responsibility

Recognition of the individual's/group's power and potential to affect others

Benefits

As symbols of individual power and as potential to advance (an increase in one's own knowledge and skills, for example)

Loyalty

Individuals may have more than one loyalty, but the direction of loyalty may change (as when, for example, one becomes integrated into a team). Loyalty in groups also has the potential to influence outcomes.

Status

Position within a hierarchy, or origin of a group or individual

Distance

Degree of acceptance of another's right to take decisions or benefit personally

We also arrived at three propositions about stakeholder perspectives which we proposed to testas we examined data:

1. Stakeholder perspectives defined by power relations offer more insights into evaluations than definitions based on job or position.

2. Stakeholder perspectives defined by power relations will have greater explanatory potential than considerations of cross-cultural differences when examining and understanding reactions to evaluation or an evaluation.

3. Understanding stakeholder perspectives will enable us to plan and organise evaluations more effectively, and to promote a greater and better use of their findings.

The first proposition should be self-evident in the light of the preceding discussion. The second proposition is relevant in development, and derives from an earlier study suggesting that the existence of an evaluation culture reveals more about the utilisation of evaluation than attempts to explain utilisation through cross-cultural difference (Murphy 1997). The third proposition follows from the first two and would therefore be true for any approach.

4 Stakeholder perspectives

Our data come from an as yet small number of interviews and questionnaire responses from representatives of different stakeholder groups, which include funding agencies, evaluators, project managers and teachers. Other data used come from project reports and field notes of our own, and these were used in deriving the above list of areas. For reasons of space we will not give any more details about the design of the survey. Also, the categories considered here do not include all those which have been previously listed because we do not have adequate data to justify those which have been omitted.

4.1 Knowledge about the project or the evaluation is expressed in a number of ways.

You need workshops to get people involved and so they can understand.

This quote from an evaluation contractor identifies professional knowledge as a precondition for getting stakeholders involved, that is, being able to exercise power. It is an interest of the contractor to get this to happen and the contractor's belief is that it will promote ownership and favour project sustainability. The simultaneous passing on of control and responsibility is not perceived as a threat, a point which seems to support Guba and Lincoln's (1989: 267) idea that power is not to be shared out in a restructuring that aims to empower, but grown (new power is created).

The consultative nature of the partnership made acceptance of evaluation by local stakeholders easier.

This remark by a project stakeholder after an evaluation suggests that open communication about knowledge where the stakeholders are information users (cf. Patton 1997) means that the latter are more likely to use their power to utilise the evaluation. Their power, in addition, has been acknowledged. The following remark from the contractor supports this line of procedure, presumably because the expectation is that it responds to the interests of more stakeholders and encourages them to use their power:

I'd like all parties to understand the nature of evaluation, to have seen the TORs, to have had a hand in drawing them up, know who its for, what's to be done, what the implications are. It should be an open relationship.

Such comments beg the further question as to the nature of the consultation, involvement and partnership. To what extent is this realised through mere information exchange? To what extent are the participants in an evaluation actually enfranchised or empowered by the process? To what extent are they in a position to influence events at the various stages of an evaluation process? This in turn raises questions about the nature of expertise.

4.2 Expertise includes dimensions of learning and understanding, and these issues were raised by several respondents.

PE [Partnership Evaluation] is meant to be a positive experience for both sides, and a learning experience

Everyone involved should be learning, because there is shared ownership...

There is a trade off between learning to evaluate and quality of conclusions.

Each group learned from the other.

How about building in some kind of attachment that will allow the Fifo [fly-in fly-out evaluator] to work with/train personnel?

These observations from three stakeholder groups – contractor, evaluation manager, evaluation participant – reinforce the perception that learning to evaluate is important and that it empowers those who learn (Kiely et al 1995, Murphy 1996). Comment (3) – from an evaluator contractually engaged in carrying out an evaluation – introduces the inevitable tension between the learning process on the one hand and the dimension of accountability on the other.

At this point, we may ask, At the end of the day, which is more important: the learning or the integrity and quality of the evaluation findings and report? These are not easy questions to answer and clearly concern different stakeholder interests. Nonetheless, if we find ourselves working in a climate of partnership evaluations, then greater clarity about our own accountability relationships (as evaluators) with a funding agency and/or with the project community is required. This clarity is crucial for all stakeholders involved, since different interests need to be identified and satisfied.

4.3 Consideration of issues of control

Consideration of control raises questions about the conditions that would need to be in place in order for some balance of control to operate amongst the participants in an evaluation.

The period of serious work by locals should be included in their annual work targets.

Time is a constraint. School time is strictly for teaching and little is spent on evaluation of projects.


The points raised here are expressed as concern with time and they link with issues about levels of responsibility, extent of involvement and, presumably, ownership. In our terms, the issue here is about power - the power to act -because, at present, someone else's power to oblige these people to do other things apparently precludes their involvement in evaluation.

4.4 Consideration of status

Status is defined here in terms of an individual's position in a hierarchy -project, ministry or institution. The comments we gathered were very much to do with evaluators' status and mode of operating, in other words, how they exercise their power:

At one end of the scale there were evaluators who were a bit dictatorial while at the other end there were those who were empathetic.

This, of course, suggests the need to consider power style because this respondent is referring to experience in one project with different evaluators.

With reference to experienced and more senior teachers the following was mentioned:

Lots can be improved, tapped from focused discussions.

The evaluator should, in fact, get these teachers to reflect on what they have been doing and to evaluate themselves.

There is a strain created so it becomes a one-way discussion thereafter.

I would recommend that evaluation findings be effected in a way that will be beneficial to the project...

What emerges from these data is that, unsurprisingly, there are differences of interest between the stakeholder groups and, again, a perception of those with higher status using their power in ways which are not accepted. These respondents, in other words, do not accept the implied power distance.

In terms of project management and promoting dialogue within an evaluation framework, it would appear there are indications here that insights can be gained from gathering information about the different prevailing interests and power relations in order to understand the stakeholders' various perspectives. Alongside the differences there are themes of concern to more than one target group. These are tentative conclusions as much more work needs to be done to develop critical examination of the three propositions. Interestingly, however, the majority of issues raised in our data so far do have implications for the ways in which evaluations, in particular partnership evaluations, are managed. We now conclude with some of these implications.

5 Implications for managing evaluation

The ideas we list here are not new, and have appeared before in discussions of the principles of educational management and of managing evaluation (e.g. Aspinwall et al 1992, Everard & Morris 1996). The only value we would claim for revisiting them afresh while doing participatory evaluations is just that they come with new empirical support. We suggest that evaluators planning to do participatory research should:

· plan for open communication.
· define what partnership evaluation is to mean in the context.
· put power/responsibility at the level where decisions will be most effectively taken.
· resource time to learn to evaluate and to participate in evaluations.

To this list we propose tentatively to add that evaluators should:

· identify stakeholder interests.
· identify power relations between stakeholder groups.

Footnote

1. This perception of the situation appears to be limited since there is a lot of evidence to counter such a rosy interpretation of the scene – through talking to senior figures rather than practitioners (Mthembu 1996).