Chapter 3 - Research methodology and project descriptions

The International Drinking Water Supply and Sanitation Decade (IDWSSD) brought investment in hundreds of small and large projects. This study takes advantage of the rich data base available in the evaluation reports of agencies involved in the IDWSSD. It is based on analyses of 121 completed rural water supply projects executed by eighteen different agencies in Asia, Africa, and Latin America; the resulting hypotheses were followed by more specific testing in selected countries. The primary methodological advance of this study is that it uses a large number of case studies and cases, converts them into ratings on a range of variables, and statistically tests a model specifying the relationship of participation to project effectiveness and local capacity building as well as to empowerment.

Carrying out this study required contacting many of the major international support agencies, NGOs, water and sanitation reference centers, and libraries around the world. In addition, regional and country offices of the United Nations Development Programme and the World Bank Water and Sanitation Program collected reports. Out of an initial listing of over 800 references, about 400 reports were received and screened to determine whether they were appropriate subjects for analysis.

Many of the evaluation reports dealt with policies, in-depth studies, and institution-building activities. Projects had to meet two criteria to be included in the study: the focus of the project had to be physical implementation of water supply, and the report needed to have sufficient information to permit analysis.

The evaluation reports were based on three- to five-week impact assessments made by teams of two to five experienced evaluators who had conducted field visits. The teams used a variety of data sources, including extensive interviews and other field visits with staff and beneficiaries, workshops, and reviews of project documentation.

Reports were not screened for participatory activity in the projects, because the quality of in-depth information on participation was uneven (understandably so, since the objectives of the evaluation reports differ from those of the present research). Often the reporting on management and community-level practices was less extensive than the reporting on outputs, inputs, institutional relationships, and external context. The reports were generally sketchy on the costs, details of the organization of the local water system, and role of women. Fortunately, project evaluation reports were supplemented in many cases by anthropological research studies and socioeconomic surveys done within the project context either during the planning stages or in analyses of impact.

Methodology

Within the noneconomic social sciences there has been a long tradition of using qualitative data and small numbers of in-depth case studies. Sociologists, psychologists, and anthropologists have increasingly turned to quantification of essentially qualitative data obtained through open-ended exploration of issues, however, as a way to subject their hypotheses to statistical tests and to explore the causal relations among variables. One of the key methodological differences between economics and the other social sciences is the process used in arriving at the numbers. Anthropologists, for instance, start with an open-ended process, explore the universe of possibilities, and do not immediately use structured surveys and questionnaires to get at the numbers. Structuring and ordering of variables are inductive and done much later in the process of inquiry than they are in economics.

This study combines model testing, using multivariate analysis of data, with in-depth qualitative analysis of particular cases. The study draws particularly upon and adapts methodologies from two research projects, one by Milton Esman and Norman Uphoff (1984) and the other by Kurt Finsterbusch and Warren Van Wicklin (1987).

A research project supported by USAID (U.S. Agency for International Development) was undertaken in 1975 to clarify the concept of participation; it led to a five-year effort to operationalize the concept and understand its limits, potentials, and applications. Many applied studies grew out of collaborative in-depth work in six countries. The desire for even more quantified, comparative assessments eventually produced some innovative efforts in methodology and a systematic research project on what determined the performance of local membership organizations (Esman and Uphoff 1984).

Realizing the limitations of small samples and purely qualitative assessments, Esman and Uphoff decided to do a large-scale, quantified comparative analysis. They proceeded to identify 150 case studies in the literature that contained enough information to evaluate at least most of fifty-five independent and dependent (performance) variables. Research assistants summarized all of the qualitative and quantitative data from the case studies onto a standard protocol, which was scored for the different variables, using standardized, specified criteria. To test whether this sample was large enough for analytic purposes, the sample was broken into three groups of fifty, each of which was scored. When the subsamples were compared for all variables by nonparametric statistical tests, the authors found statistically significant differences for less than 5 percent of the variables, which is equivalent to meeting a test of significance at the 95 percent confidence level. These results satisfied the researchers that enlarging the sample by another fifty cases would not significantly change the variance of the data being analyzed.

The large sample size allowed study of a multiplicity of variables, the interactions among them, and their effect on local organization performance. Variables fell into five categories: environmental, structural, functional, and participation variables, and exogenous factors.

The major challenge for Esman and Uphoff was to quantify factors and subfactors so that each of them could be rated with a high degree of reliability by trained researchers. Achieving equivalence of meaning across cultures made the task more difficult. After extensive consultations, the project leaders developed a system of coding for fifty-five variables on a 5-point scale. Pearson-product moment correlations were used to test the relationships among variables; regression analysis identified the main factors contributing to the largest variance in performance. The study was also informed by the published literature on participation and primary data collection on that topic in selected countries. Although the sample was not random and the scoring of factors was essentially subjective, the project yielded useful patterns on a subject that had not previously been studied quantitatively. The results of this project were published in 1984 by Esman and Uphoff. Local Organizations: Intermediaries in Rural Development, the published results of this project, remains a milestone in the literature on participation.

The study found relatively high correlations between effective task performance and the participatory orientation of the organization. Tasks included planning and goal setting, conflict management, resource mobilization and management, service provision, and claim making. The authors also found that under adverse conditions, local organizations experience less-than-average success unless they are highly participatory and have effective leadership.

The methodology outlined above, which attempts to explore statistically the relationship among variables affecting the performance of local organizations-including participation- was further refined by Kurt Finsterbusch and Warren Van Wicklin (1987). Rather than focus on local organization, Finsterbusch and Van Wicklin analyzed factors affecting project performance; they included participation as one of the variables. An input-output framework guided their analysis. Their model disaggregates input, process, output, impact, and contextual factors.

The study by Finsterbusch and Van Wicklin was based on analyses of evaluation reports for fifty-two USAID projects across sectors. The reports followed a standard format, making it easy to develop and apply a coding system. Construction projects constituted most of the sample, but some health, education, and agricultural research projects were also included.

The study found only moderate, but statistically significant, correlations between participation and project effectiveness. The authors found that effective projects involved good communication from the project authorities to the public; the projects were well received by the beneficiaries, but they were not always participatory.

The main problem with the conclusions of Finsterbusch and Van Wicklin is that their study was based on a relatively small sample and included many different types of projects that do not require beneficiary participation to be successful. The findings therefore may reflect the nature of the projects chosen rather than the importance of participation in its own right.

To follow up on the hypothesis that beneficiary participation is not equally critical in all projects, Finsterbusch and Van Wicklin expanded the sample size to seventy-one projects and did further analyses by type of project. They found that the correlations with participation were much lower for education and health projects (personal communication). The expanded study still covered a range of USAID-funded projects.

Methodology of the Current Study

This study borrows from the methodology described for the two studies cited above, using the conceptual framework developed in Chapter 2; the aim is to test the contribution of participation to the effectiveness and capacity building aspects of rural water supply projects. The main issues of interest were listed and a few questions devised to address each issue. The preliminary questionnaire thus developed consisted of over 200 different items. Based on interviews and feedback from sector specialists, review of the literature, consideration of the information available in reports, and a pilot run on four cases, the questionnaire was eventually reduced to 145 items. Multiple items were used to address one concept. Most of the items were rated on a scale of 0 through 7, with some of the key performance variables scored on a scale of 0 through 10. A code book was developed, which specified the characteristics to be considered in assigning a particular score for each variable.

Two coders (one male and one female) were trained to apply the questionnaire to each project report to produce subjective-but cardinal-ratings until there was a high level of agreement between the coders. The coding of each report took between three and four hours, on average. Initially the two coders met periodically to compare ratings and discuss differences in ratings. These discussions ensured that the two coders were using the same criteria to arrive at ratings and that particular pieces of information had not been overlooked.

Data Limitations

Although every care was taken to avoid ambiguity to and increase coder reliability through training, the ratings were subjective and scores do differ between coders. Since the scores were subjective, coders could agree in general but still assign different numerical values to a particular variable. However, as long as a particular coder was consistent, the difference between coders did not affect correlation analyses. On the other hand, if there are differences in direction of rating between coders, the reliability of the rating of that variable is called into question.

For this reason, two indicators of data reliability were built into the process of data analysis. The first indicator is the intercoder correlation coefficients for the two sets of coded scores. Most of the coefficients were higher than 0.85. The key variables of interest in the study had intercoder coefficients of 0.9 or above (overall effectiveness, 0.96; overall beneficiary participation, 0.92; overall community empowerment, 0.90).

The quality of the reports was uneven, and information for some of the factors was missing. Coders were therefore asked to assign a confidence score for each variable as a second measure of reliability. Each coder indicated his or her level of confidence in the score assigned to each variable on a scale of 1 to 5. Most of these scores, as well as the confidence scores for the key variables, were high.

Subjective ratings are open to the criticism that the "halo effect" colors results. A coder aware of the hypotheses to be tested is more likely to score projects high in participation when the projects are highly effective, and vice versa-in other words, all good things go together. To check for the halo effect, regression analyses were conducted, using the input scores of one coder with the output scores of the other coder. A substantial drop in the correlation coefficients and a change in the results would indicate a significant halo effect in operation. Regressions were repeated, using nonstandardized and standardized data. The findings establish that the halo effect does not significantly alter the results of the coders. (Some findings are reported in Chapter 4. For further information see Isham, Narayan, and Pritchett 1994.)

The rest of this chapter describes in some detail the variables included in the model for the study.

Measuring Variables

As mentioned before, the questionnaire used to code the project reports contained 145 items. Based on the number of observations per item, nature of scale (that is, whether the scale was continuous), intercoder reliability, coder confidence, and analysis of principal components, the number of items included in the model was reduced to fifty. These variables are discussed fully, following a description of some background characteristics of the project, most of which are not included in the model.

Background Characteristics

The 121 projects included in the study were located in 49 different countries. Forty-eight percent of the projects were carried out in Africa, 33 percent in Asia, and 20 percent in Latin America (table 3.1). Most of the projects were completed in the 1980s. Overall, 56 percent of the projects received some bilateral financing; 26 percent got multilateral financing; and 15 percent were financed by national and international NGOs; the remaining 3 percent were completely financed by national governments. Hence, the sample was heavily biased toward externally financed projects.

Three of the 50 items scored-total costs, population reached, and total number of project staff-give some idea of size of projects. The costs of the projects ranged from $500,000 to $250 million; 9 percent of the projects cost more than $25 million; and 17 percent cost less than $1 million.² Twenty percent of the projects reached more than 500,000 people; 47 percent, from 60,000 to 500,000; and 33 percent reached fewer than 60,000. Only 76 projects provided data on the number of staff involved; approximately 38 percent of these projects had more than 200 staff members.

Table 3.1. Background characteristics of 121 water projects

		Frequency
Variable	Category	Number	Percent
Region	Africa	58	48
	Asia	39	32
	Latin America	24	20
Year project ended	1972-80	14	12
	1981-85	25	21
	1986-90	72	60
	1991-92	10	7
Main donor	Multilateral donors	32	26
	Bilateral donors	68	56
	NGOs	18	15
	Government-financed	3	3
Cost of project	$0 3 million	39	37
	$3 million-10 million	33	32
	S10+ million	33	31
Population reached	1,000-60,000	38	33
	60,000-500,000	55	47
	500,000+	24	20
Type of technology	Protected springs	2	1
	Dug/shallow wells	8	7
	Tube wells with handpumps	26	21
	Deep-dug wells	25	20
	Gravity systems	41	34
	Power-pumped systems	20	17
Type of distribution	Spring	2	2
system	Well-head handpump	65	53
	Community standpipes	31	25
	Private connections	24	20
Per capita country	$120-340	52	46
GNP in 1989	$360 650	32	28
	$710-1,000	13	11
	$1,010-1,760	17	15
Average annual	S0-199	81	67
village income	$200-399	16	13
	$400-599	14	12
	$600+	10	8

In keeping with the focus of the study on one subsector, only projects whose primary objective was implementation of rural water systems were selected. However, since donors often promulgate an integrated approach to water systems, many projects included components for sanitation construction and other primary health care activities (49 percent). Another 7 percent of projects included income-generation activities.

Technology type. The projects included a range of technologies, from spring captures to power-pumped, piped-water systems. Overall, 36 percent of the projects installed just one type of technology, 46 percent used two or three different technology systems, and 18 percent installed four or more different types of systems. Twenty percent of the projects included household connections; 25 percent, community standposts; and 53 percent, handpumps. The remainder were spring captures or rain tanks. Only in 10 percent of the cases were communities given any choice of technology.

Income. Three items focused on income levels: per capita gross national product (GNP), per capita client income, and village income. Clearly, projects were targeted to the poorest countries: approximately 75 percent of the projects were from countries with annual per capita GNP of less than $650. Intercoder reliability on client and village income was low, particularly for the type of client most served. Hence, unfortunately, these two measures had to be excluded from further analysis.

Measuring the Variables Included in the Model

The model groups variables in six categories: (1) performance outcomes; (2) beneficiary participation; (3) nonparticipation determinants; (4) proximate determinants of outcomes; (5) determinants of participation; and (6) intermediate steps to participation. Each category is discussed below.

Performance outcomes. Performance outcomes were of two types, water-related outcomes and capacity building (or empowerment) outcomes. (See table 3.2 for a summary of the outcome variables that were included in the model.)

Water-related outcomes covered six performance variables. Overall project effectiveness is the key outcome variable used in the study; it was rated on a scale of 0 through 10. Overall effectiveness measured all project costs and benefits in the areas of construction, operations and maintenance (O&M), health and sanitation education, extension and community development, institutional development, and income generation. Several maintenance measures were coded and one was selected for inclusion in the model: percentage of water systems functioning and in good condition. The measure addressed only downtime after breakdown during normal times, which excluded natural calamities such as floods and drought.

Access to and use of new or improved water systems can produce several benefits, namely, time savings, other income-generating activities, and improved health. The economic benefits variable encompassed all of these measures. Coverage achieved by improved water systems-that is, percentage of the target population who used the improved system-was included as another performance indicator in the analysis.

Two other variables of a slightly different nature, equality of access (whether everyone had equal access to the water system) and environmental effects, were also included. Although drinking water projects are generally not expected to have a significant environmental impact, they do have the potential to improve reforestation or greenery or to adversely affect the environment by waterlogging the area immediately surrounding the project.

Table 3.2. Measures of performance outcomes and participation

Performance outcomes
		Water-related outcomes
			Overall project effectiveness
			Percentage of water systems functioning and in good condition
			Economic benefits
			Percentage of target population reached
			Equality of access
			Environmental effects
		Capacity building outcomes
			Community empowerment
			Women's empowerment
			Capacity and skills related to water systems
			Strengthened local organizations
			Strengthened local leaders
			Beneficiary participation
			Overall beneficiary participation
			Overall women's participation
Participation in design
		Women's participation in design
		Participation in construction
		Women's participation in construction
		Participation in operation and maintenance
		Women's participation in operation and maintenance

Local capacity is essential to achieving sustainability, so five capacity building (or empowerment) outcomes at the individual level and at the group or organizational level were chosen for inclusion: overall beneficiary empowerment, women's empowerment, increased skills in water-related tasks, strengthened local organizations, and strengthened local leaders.

Beneficiary participation. Several different measures of participation were selected for the model. The lower end of the scale represented information sharing; at the higher end were decisionmaking and control. The measures included an indicator of overall participation of beneficiaries in all aspects and stages of the project. The other measures focused on gender and participation during a particular stage of the project.

Nonparticipation determinants of outcomes. Obviously, beneficiary participation is not the only determinant of project outcomes. A range of exogenous variables affects outcomes directly, and another category of variables affects outcomes either directly or indirectly through participation. Table 3.3 lists these determinants of outcomes.

There were seven variables in the first set of direct determinants, primarily focusing on general project characteristics and per capita GNP. The second set of direct and indirect variables were grouped into five subcategories: technology, external agents, client characteristics, external climate, and management.

Proximate deteminants of outcomes. This set of variables included those factors through which, it was hypothesized, beneficiary participation worked to effect outcomes. In a sense, then, these variables are intermediate ones on the way to determining final outcomes. Variables in this category were classified as institutional or physical. Institutional variables covered the quality of project design and quality of implementation. Physical variables included the quality of construction and operations and maintenance. Since O&M is such a major issue, two other intermediate outputs were added to the model: maintenance after one year and maintenance after five years.

Participation determinants. Overall beneficiary participation is determined by characteristics of the beneficiaries and the agency. (See table 3.4 for a listing of the determinants and elements of participation.) If both beneficiaries and agencies perceive the net benefits of participation to be high, participation can occur. An overall measure of perceived net benefits was developed after coders separately rated agency and beneficiary costs and benefits.

Net benefits are a function of client and agency characteristics, which themselves are determinants of overall participation. Client characteristics included the demand of clients, or the commitment clients made before implementation of the project; the skills and knowledge of clients; the quality of broad-based leadership; the dependence on strong leaders; the social organization of clients to undertake water-related tasks; and the extent to which organization builds upon local traditions and structures. Important agency characteristics are use of local knowledge, degree to which beneficiary participation was made a goal, implementation flexibility, autonomy of an agency to manage its own affairs, consensus on objectives and means, and the degree to which the project was driven by physical targets.

Table 3.3. Non-participation determinants of outcomes

Direct determinants
	Total cost
	Project complexity
	Adequacy of facilities and equipment
	Difficulty in recruiting and retaining staff
	Availability of spare parts and technicians
	Extent objectives clearly specified
	Per capita GNP
Direct and indirect determinants
	Technology
		Type of technology (sophistication of technology)
		Appropriateness of technology
	External agents
		Support of host government
		Understanding between agencies
	Client characteristics
		Average number of users per system
		Presence of other water sources
		Village income
	External climate
		Conduciveness of political climate
		Conduciveness of economic climate
		Conduciveness of sociocultural climate
		Conduciveness of geological environment
	Management
		Overall quality of management
		Skills of staff
Proximate determinants
	Institutional outputs
		Quality of project design
		Quality of project implementation
	Physical outputs
		Quality of construction
		Adequacy of operation and maintenance
		Maintenance after one year
		Maintenance after five years

Table 3.4. Determinants of participation

Determinants of participation
	Client characteristics
		Commitment of clients before implementation
		Skills and knowledge
		Quality of broad-based leadership
		Dependence on strong leaders
		Extent of organization of clients
		Extent to which organization is based on traditional structure
	Agency characteristics
		Use of local knowledge
		Extent to which participation is made a goal
		Implementation flexibility
		Autonomy of project and agency
		Consensus on objectives
		Degree to which the project is driven by physical targets
	Net benefits of participation
Elements of participation
	Agency-user relations
		Responsiveness of agency to clients
		Extent to which clients listen to agents
	User voice and exit
		Extent to which control and ownership became local
		Extent to which clients exit
		Dissatisfaction of clients
	User investment in costs
		Capital investment made by clients
		Percentage of recurrent costs paid by users

Critical elements of the participation process. Participation is an iterative process; determinants of participation act through intermediate steps, which together lead toward high levels of overall participation. These steps are important milestones that arise as beneficiaries organize and as agencies put participatory plans of implementation into practice. The more important of these milestones are the responsiveness of agencies to client feedback; the extent to which clients listen to field agents; the extent to which local groups begin to gain control and ownership of resources; the users' dissatisfaction with or exit from the system; and the investment in capital and recurrent costs made by beneficiaries.

Schematic Presentation of the Model

Figure 3.1 is a schematic presentation of the model outlined above. The fundamental relationship of interest is between participation and outcomes (which are either water-related or more general capacity building outcomes). The model also posits nonparticipation factors that have an impact on outcomes. These are of two kinds: direct nonparticipation factors which influence outcomes independently, and direct and indirect nonparticipation factors which exert their influence via the participation process. Finally, the model includes participation determinants, factors that lie behind and cause participation.

Figure 3.1. Model of the relationship of participation to outcomes and other factors

To account for some of the complexity and iterations of the processes at work, the model also incorporates two intermediate relationships. The first concerns direct nonparticipation determinants, working through a series of proximate determinants of outcomes. Similarly, the determinants of participation are assumed to work through what is called "critical elements of participation" in figure 3.1.

The model and the variables associated with each are elaborated in chapter 4 (which discusses the part of the model relating to outcomes) and chapter 5 (which addresses the part of the model relating to determinants of participation).

Other Notes on Research for the Model

Statistical analysis was conducted in phases. Frequencies, cross-tabulations, correlations,³ factor analyses, and a limited number of multivariate regression analyses⁴ were performed after checking the quality of the data. The final round of data analysis, after item reduction,⁵ consisted primarily of multivariate regression analysis for model testing.

In addition, results of this study were compared to those from other studies following an extensive review of the literature, and several reviews and evaluations of projects received after completion of the coding process were considered. Findings from the review of literature and these additional evaluation reports are highlighted when relevant.