Cover Image
close this bookContribution of People's Participation: Evidence from 121 Rural Water Supply Projects (World Bank, 1995)
View the document(introduction...)
View the documentEnvironmentally sustainable development series
View the documentForeword
View the documentAcknowledgments
View the documentExecutive summary
View the documentChapter 1 - Introduction
View the documentChapter 2 - The concept of participation
View the documentChapter 3 - Research methodology and project descriptions
View the documentChapter 4 - Role of beneficiary participation in project effectiveness
View the documentChapter 5 - Factors affecting beneficiary participation
View the documentChapter 6 - Translating lessons into design features
View the documentChapter 7 - Conclusions and recommendations
Open this folder and view contentsAppendixes
View the documentNotes
View the documentBibliography

Chapter 3 - Research methodology and project descriptions

The International Drinking Water Supply and Sanitation Decade (IDWSSD) brought investment in hundreds of small and large projects. This study takes advantage of the rich data base available in the evaluation reports of agencies involved in the IDWSSD. It is based on analyses of 121 completed rural water supply projects executed by eighteen different agencies in Asia, Africa, and Latin America; the resulting hypotheses were followed by more specific testing in selected countries. The primary methodological advance of this study is that it uses a large number of case studies and cases, converts them into ratings on a range of variables, and statistically tests a model specifying the relationship of participation to project effectiveness and local capacity building as well as to empowerment.

Carrying out this study required contacting many of the major international support agencies, NGOs, water and sanitation reference centers, and libraries around the world. In addition, regional and country offices of the United Nations Development Programme and the World Bank Water and Sanitation Program collected reports. Out of an initial listing of over 800 references, about 400 reports were received and screened to determine whether they were appropriate subjects for analysis.

Many of the evaluation reports dealt with policies, in-depth studies, and institution-building activities. Projects had to meet two criteria to be included in the study: the focus of the project had to be physical implementation of water supply, and the report needed to have sufficient information to permit analysis.

The evaluation reports were based on three- to five-week impact assessments made by teams of two to five experienced evaluators who had conducted field visits. The teams used a variety of data sources, including extensive interviews and other field visits with staff and beneficiaries, workshops, and reviews of project documentation.

Reports were not screened for participatory activity in the projects, because the quality of in-depth information on participation was uneven (understandably so, since the objectives of the evaluation reports differ from those of the present research). Often the reporting on management and community-level practices was less extensive than the reporting on outputs, inputs, institutional relationships, and external context. The reports were generally sketchy on the costs, details of the organization of the local water system, and role of women. Fortunately, project evaluation reports were supplemented in many cases by anthropological research studies and socioeconomic surveys done within the project context either during the planning stages or in analyses of impact.

Methodology

Within the noneconomic social sciences there has been a long tradition of using qualitative data and small numbers of in-depth case studies. Sociologists, psychologists, and anthropologists have increasingly turned to quantification of essentially qualitative data obtained through open-ended exploration of issues, however, as a way to subject their hypotheses to statistical tests and to explore the causal relations among variables. One of the key methodological differences between economics and the other social sciences is the process used in arriving at the numbers. Anthropologists, for instance, start with an open-ended process, explore the universe of possibilities, and do not immediately use structured surveys and questionnaires to get at the numbers. Structuring and ordering of variables are inductive and done much later in the process of inquiry than they are in economics.

This study combines model testing, using multivariate analysis of data, with in-depth qualitative analysis of particular cases. The study draws particularly upon and adapts methodologies from two research projects, one by Milton Esman and Norman Uphoff (1984) and the other by Kurt Finsterbusch and Warren Van Wicklin (1987).

A research project supported by USAID (U.S. Agency for International Development) was undertaken in 1975 to clarify the concept of participation; it led to a five-year effort to operationalize the concept and understand its limits, potentials, and applications. Many applied studies grew out of collaborative in-depth work in six countries. The desire for even more quantified, comparative assessments eventually produced some innovative efforts in methodology and a systematic research project on what determined the performance of local membership organizations (Esman and Uphoff 1984).

Realizing the limitations of small samples and purely qualitative assessments, Esman and Uphoff decided to do a large-scale, quantified comparative analysis. They proceeded to identify 150 case studies in the literature that contained enough information to evaluate at least most of fifty-five independent and dependent (performance) variables. Research assistants summarized all of the qualitative and quantitative data from the case studies onto a standard protocol, which was scored for the different variables, using standardized, specified criteria. To test whether this sample was large enough for analytic purposes, the sample was broken into three groups of fifty, each of which was scored. When the subsamples were compared for all variables by nonparametric statistical tests, the authors found statistically significant differences for less than 5 percent of the variables, which is equivalent to meeting a test of significance at the 95 percent confidence level. These results satisfied the researchers that enlarging the sample by another fifty cases would not significantly change the variance of the data being analyzed.

The large sample size allowed study of a multiplicity of variables, the interactions among them, and their effect on local organization performance. Variables fell into five categories: environmental, structural, functional, and participation variables, and exogenous factors.

The major challenge for Esman and Uphoff was to quantify factors and subfactors so that each of them could be rated with a high degree of reliability by trained researchers. Achieving equivalence of meaning across cultures made the task more difficult. After extensive consultations, the project leaders developed a system of coding for fifty-five variables on a 5-point scale. Pearson-product moment correlations were used to test the relationships among variables; regression analysis identified the main factors contributing to the largest variance in performance. The study was also informed by the published literature on participation and primary data collection on that topic in selected countries. Although the sample was not random and the scoring of factors was essentially subjective, the project yielded useful patterns on a subject that had not previously been studied quantitatively. The results of this project were published in 1984 by Esman and Uphoff. Local Organizations: Intermediaries in Rural Development, the published results of this project, remains a milestone in the literature on participation.

The study found relatively high correlations between effective task performance and the participatory orientation of the organization. Tasks included planning and goal setting, conflict management, resource mobilization and management, service provision, and claim making. The authors also found that under adverse conditions, local organizations experience less-than-average success unless they are highly participatory and have effective leadership.

The methodology outlined above, which attempts to explore statistically the relationship among variables affecting the performance of local organizations-including participation- was further refined by Kurt Finsterbusch and Warren Van Wicklin (1987). Rather than focus on local organization, Finsterbusch and Van Wicklin analyzed factors affecting project performance; they included participation as one of the variables. An input-output framework guided their analysis. Their model disaggregates input, process, output, impact, and contextual factors.

The study by Finsterbusch and Van Wicklin was based on analyses of evaluation reports for fifty-two USAID projects across sectors. The reports followed a standard format, making it easy to develop and apply a coding system. Construction projects constituted most of the sample, but some health, education, and agricultural research projects were also included.

The study found only moderate, but statistically significant, correlations between participation and project effectiveness. The authors found that effective projects involved good communication from the project authorities to the public; the projects were well received by the beneficiaries, but they were not always participatory.

The main problem with the conclusions of Finsterbusch and Van Wicklin is that their study was based on a relatively small sample and included many different types of projects that do not require beneficiary participation to be successful. The findings therefore may reflect the nature of the projects chosen rather than the importance of participation in its own right.

To follow up on the hypothesis that beneficiary participation is not equally critical in all projects, Finsterbusch and Van Wicklin expanded the sample size to seventy-one projects and did further analyses by type of project. They found that the correlations with participation were much lower for education and health projects (personal communication). The expanded study still covered a range of USAID-funded projects.

Methodology of the Current Study

This study borrows from the methodology described for the two studies cited above, using the conceptual framework developed in Chapter 2; the aim is to test the contribution of participation to the effectiveness and capacity building aspects of rural water supply projects. The main issues of interest were listed and a few questions devised to address each issue. The preliminary questionnaire thus developed consisted of over 200 different items. Based on interviews and feedback from sector specialists, review of the literature, consideration of the information available in reports, and a pilot run on four cases, the questionnaire was eventually reduced to 145 items. Multiple items were used to address one concept. Most of the items were rated on a scale of 0 through 7, with some of the key performance variables scored on a scale of 0 through 10. A code book was developed, which specified the characteristics to be considered in assigning a particular score for each variable.

Two coders (one male and one female) were trained to apply the questionnaire to each project report to produce subjective-but cardinal-ratings until there was a high level of agreement between the coders. The coding of each report took between three and four hours, on average. Initially the two coders met periodically to compare ratings and discuss differences in ratings. These discussions ensured that the two coders were using the same criteria to arrive at ratings and that particular pieces of information had not been overlooked.

Data Limitations

Although every care was taken to avoid ambiguity to and increase coder reliability through training, the ratings were subjective and scores do differ between coders. Since the scores were subjective, coders could agree in general but still assign different numerical values to a particular variable. However, as long as a particular coder was consistent, the difference between coders did not affect correlation analyses. On the other hand, if there are differences in direction of rating between coders, the reliability of the rating of that variable is called into question.

For this reason, two indicators of data reliability were built into the process of data analysis. The first indicator is the intercoder correlation coefficients for the two sets of coded scores. Most of the coefficients were higher than 0.85. The key variables of interest in the study had intercoder coefficients of 0.9 or above (overall effectiveness, 0.96; overall beneficiary participation, 0.92; overall community empowerment, 0.90).

The quality of the reports was uneven, and information for some of the factors was missing. Coders were therefore asked to assign a confidence score for each variable as a second measure of reliability. Each coder indicated his or her level of confidence in the score assigned to each variable on a scale of 1 to 5. Most of these scores, as well as the confidence scores for the key variables, were high.

Subjective ratings are open to the criticism that the "halo effect" colors results. A coder aware of the hypotheses to be tested is more likely to score projects high in participation when the projects are highly effective, and vice versa-in other words, all good things go together. To check for the halo effect, regression analyses were conducted, using the input scores of one coder with the output scores of the other coder. A substantial drop in the correlation coefficients and a change in the results would indicate a significant halo effect in operation. Regressions were repeated, using nonstandardized and standardized data. The findings establish that the halo effect does not significantly alter the results of the coders. (Some findings are reported in Chapter 4. For further information see Isham, Narayan, and Pritchett 1994.)

The rest of this chapter describes in some detail the variables included in the model for the study.

Measuring Variables

As mentioned before, the questionnaire used to code the project reports contained 145 items. Based on the number of observations per item, nature of scale (that is, whether the scale was continuous), intercoder reliability, coder confidence, and analysis of principal components, the number of items included in the model was reduced to fifty. These variables are discussed fully, following a description of some background characteristics of the project, most of which are not included in the model.

Background Characteristics

The 121 projects included in the study were located in 49 different countries. Forty-eight percent of the projects were carried out in Africa, 33 percent in Asia, and 20 percent in Latin America (table 3.1). Most of the projects were completed in the 1980s. Overall, 56 percent of the projects received some bilateral financing; 26 percent got multilateral financing; and 15 percent were financed by national and international NGOs; the remaining 3 percent were completely financed by national governments. Hence, the sample was heavily biased toward externally financed projects.

Three of the 50 items scored-total costs, population reached, and total number of project staff-give some idea of size of projects. The costs of the projects ranged from $500,000 to $250 million; 9 percent of the projects cost more than $25 million; and 17 percent cost less than $1 million.2 Twenty percent of the projects reached more than 500,000 people; 47 percent, from 60,000 to 500,000; and 33 percent reached fewer than 60,000. Only 76 projects provided data on the number of staff involved; approximately 38 percent of these projects had more than 200 staff members.

Table 3.1. Background characteristics of 121 water projects



Frequency

Variable

Category

Number

Percent

Region

Africa

58

48


Asia

39

32


Latin America

24

20

Year project ended

1972-80

14

12


1981-85

25

21


1986-90

72

60


1991-92

10

7

Main donor

Multilateral donors

32

26


Bilateral donors

68

56


NGOs

18

15


Government-financed

3

3

Cost of project

$0 3 million

39

37


$3 million-10 million

33

32


S10+ million

33

31

Population reached

1,000-60,000

38

33


60,000-500,000

55

47


500,000+

24

20

Type of technology

Protected springs

2

1


Dug/shallow wells

8

7


Tube wells with handpumps

26

21


Deep-dug wells

25

20


Gravity systems

41

34


Power-pumped systems

20

17

Type of distribution

Spring

2

2

system

Well-head handpump

65

53


Community standpipes

31

25


Private connections

24

20

Per capita country

$120-340

52

46

GNP in 1989

$360 650

32

28


$710-1,000

13

11


$1,010-1,760

17

15

Average annual

S0-199

81

67

village income

$200-399

16

13


$400-599

14

12


$600+

10

8

In keeping with the focus of the study on one subsector, only projects whose primary objective was implementation of rural water systems were selected. However, since donors often promulgate an integrated approach to water systems, many projects included components for sanitation construction and other primary health care activities (49 percent). Another 7 percent of projects included income-generation activities.

Technology type. The projects included a range of technologies, from spring captures to power-pumped, piped-water systems. Overall, 36 percent of the projects installed just one type of technology, 46 percent used two or three different technology systems, and 18 percent installed four or more different types of systems. Twenty percent of the projects included household connections; 25 percent, community standposts; and 53 percent, handpumps. The remainder were spring captures or rain tanks. Only in 10 percent of the cases were communities given any choice of technology.

Income. Three items focused on income levels: per capita gross national product (GNP), per capita client income, and village income. Clearly, projects were targeted to the poorest countries: approximately 75 percent of the projects were from countries with annual per capita GNP of less than $650. Intercoder reliability on client and village income was low, particularly for the type of client most served. Hence, unfortunately, these two measures had to be excluded from further analysis.

Measuring the Variables Included in the Model

The model groups variables in six categories: (1) performance outcomes; (2) beneficiary participation; (3) nonparticipation determinants; (4) proximate determinants of outcomes; (5) determinants of participation; and (6) intermediate steps to participation. Each category is discussed below.

Performance outcomes. Performance outcomes were of two types, water-related outcomes and capacity building (or empowerment) outcomes. (See table 3.2 for a summary of the outcome variables that were included in the model.)

Water-related outcomes covered six performance variables. Overall project effectiveness is the key outcome variable used in the study; it was rated on a scale of 0 through 10. Overall effectiveness measured all project costs and benefits in the areas of construction, operations and maintenance (O&M), health and sanitation education, extension and community development, institutional development, and income generation. Several maintenance measures were coded and one was selected for inclusion in the model: percentage of water systems functioning and in good condition. The measure addressed only downtime after breakdown during normal times, which excluded natural calamities such as floods and drought.

Access to and use of new or improved water systems can produce several benefits, namely, time savings, other income-generating activities, and improved health. The economic benefits variable encompassed all of these measures. Coverage achieved by improved water systems-that is, percentage of the target population who used the improved system-was included as another performance indicator in the analysis.

Two other variables of a slightly different nature, equality of access (whether everyone had equal access to the water system) and environmental effects, were also included. Although drinking water projects are generally not expected to have a significant environmental impact, they do have the potential to improve reforestation or greenery or to adversely affect the environment by waterlogging the area immediately surrounding the project.

Table 3.2. Measures of performance outcomes and participation

Performance outcomes


Water-related outcomes



Overall project effectiveness



Percentage of water systems functioning and in good condition



Economic benefits



Percentage of target population reached



Equality of access



Environmental effects


Capacity building outcomes



Community empowerment



Women's empowerment



Capacity and skills related to water systems



Strengthened local organizations



Strengthened local leaders



Beneficiary participation



Overall beneficiary participation



Overall women's participation

Participation in design


Women's participation in design


Participation in construction


Women's participation in construction


Participation in operation and maintenance


Women's participation in operation and maintenance

Local capacity is essential to achieving sustainability, so five capacity building (or empowerment) outcomes at the individual level and at the group or organizational level were chosen for inclusion: overall beneficiary empowerment, women's empowerment, increased skills in water-related tasks, strengthened local organizations, and strengthened local leaders.

Beneficiary participation. Several different measures of participation were selected for the model. The lower end of the scale represented information sharing; at the higher end were decisionmaking and control. The measures included an indicator of overall participation of beneficiaries in all aspects and stages of the project. The other measures focused on gender and participation during a particular stage of the project.

Nonparticipation determinants of outcomes. Obviously, beneficiary participation is not the only determinant of project outcomes. A range of exogenous variables affects outcomes directly, and another category of variables affects outcomes either directly or indirectly through participation. Table 3.3 lists these determinants of outcomes.

There were seven variables in the first set of direct determinants, primarily focusing on general project characteristics and per capita GNP. The second set of direct and indirect variables were grouped into five subcategories: technology, external agents, client characteristics, external climate, and management.

Proximate deteminants of outcomes. This set of variables included those factors through which, it was hypothesized, beneficiary participation worked to effect outcomes. In a sense, then, these variables are intermediate ones on the way to determining final outcomes. Variables in this category were classified as institutional or physical. Institutional variables covered the quality of project design and quality of implementation. Physical variables included the quality of construction and operations and maintenance. Since O&M is such a major issue, two other intermediate outputs were added to the model: maintenance after one year and maintenance after five years.

Participation determinants. Overall beneficiary participation is determined by characteristics of the beneficiaries and the agency. (See table 3.4 for a listing of the determinants and elements of participation.) If both beneficiaries and agencies perceive the net benefits of participation to be high, participation can occur. An overall measure of perceived net benefits was developed after coders separately rated agency and beneficiary costs and benefits.

Net benefits are a function of client and agency characteristics, which themselves are determinants of overall participation. Client characteristics included the demand of clients, or the commitment clients made before implementation of the project; the skills and knowledge of clients; the quality of broad-based leadership; the dependence on strong leaders; the social organization of clients to undertake water-related tasks; and the extent to which organization builds upon local traditions and structures. Important agency characteristics are use of local knowledge, degree to which beneficiary participation was made a goal, implementation flexibility, autonomy of an agency to manage its own affairs, consensus on objectives and means, and the degree to which the project was driven by physical targets.

Table 3.3. Non-participation determinants of outcomes

Direct determinants


Total cost


Project complexity


Adequacy of facilities and equipment


Difficulty in recruiting and retaining staff


Availability of spare parts and technicians


Extent objectives clearly specified


Per capita GNP

Direct and indirect determinants


Technology



Type of technology (sophistication of technology)



Appropriateness of technology


External agents



Support of host government



Understanding between agencies


Client characteristics



Average number of users per system



Presence of other water sources



Village income


External climate



Conduciveness of political climate



Conduciveness of economic climate



Conduciveness of sociocultural climate



Conduciveness of geological environment


Management



Overall quality of management



Skills of staff

Proximate determinants


Institutional outputs



Quality of project design



Quality of project implementation


Physical outputs



Quality of construction



Adequacy of operation and maintenance



Maintenance after one year



Maintenance after five years

Table 3.4. Determinants of participation

Determinants of participation


Client characteristics



Commitment of clients before implementation



Skills and knowledge



Quality of broad-based leadership



Dependence on strong leaders



Extent of organization of clients



Extent to which organization is based on traditional structure


Agency characteristics



Use of local knowledge



Extent to which participation is made a goal



Implementation flexibility



Autonomy of project and agency



Consensus on objectives



Degree to which the project is driven by physical targets


Net benefits of participation

Elements of participation


Agency-user relations



Responsiveness of agency to clients



Extent to which clients listen to agents


User voice and exit



Extent to which control and ownership became local



Extent to which clients exit



Dissatisfaction of clients


User investment in costs



Capital investment made by clients



Percentage of recurrent costs paid by users

Critical elements of the participation process. Participation is an iterative process; determinants of participation act through intermediate steps, which together lead toward high levels of overall participation. These steps are important milestones that arise as beneficiaries organize and as agencies put participatory plans of implementation into practice. The more important of these milestones are the responsiveness of agencies to client feedback; the extent to which clients listen to field agents; the extent to which local groups begin to gain control and ownership of resources; the users' dissatisfaction with or exit from the system; and the investment in capital and recurrent costs made by beneficiaries.

Schematic Presentation of the Model

Figure 3.1 is a schematic presentation of the model outlined above. The fundamental relationship of interest is between participation and outcomes (which are either water-related or more general capacity building outcomes). The model also posits nonparticipation factors that have an impact on outcomes. These are of two kinds: direct nonparticipation factors which influence outcomes independently, and direct and indirect nonparticipation factors which exert their influence via the participation process. Finally, the model includes participation determinants, factors that lie behind and cause participation.


Figure 3.1. Model of the relationship of participation to outcomes and other factors

To account for some of the complexity and iterations of the processes at work, the model also incorporates two intermediate relationships. The first concerns direct nonparticipation determinants, working through a series of proximate determinants of outcomes. Similarly, the determinants of participation are assumed to work through what is called "critical elements of participation" in figure 3.1.

The model and the variables associated with each are elaborated in chapter 4 (which discusses the part of the model relating to outcomes) and chapter 5 (which addresses the part of the model relating to determinants of participation).

Other Notes on Research for the Model

Statistical analysis was conducted in phases. Frequencies, cross-tabulations, correlations,3 factor analyses, and a limited number of multivariate regression analyses4 were performed after checking the quality of the data. The final round of data analysis, after item reduction,5 consisted primarily of multivariate regression analysis for model testing.

In addition, results of this study were compared to those from other studies following an extensive review of the literature, and several reviews and evaluations of projects received after completion of the coding process were considered. Findings from the review of literature and these additional evaluation reports are highlighted when relevant.