6. Evaluation of technologies

At several points in the phased evolution of the technology development and transfer process, it will be necessary to assess and evaluate potential identified technologies and a hypothetical development and transfer process. Therefore, it is necessary to have criteria for evaluating relative appropriateness of various technology development or transfer strategies in order to determine appropriately meritorious technologies and associated development and transfer strategies. There are many factors that need to be considered in doing this. Large-scale technology development consumes financial and other resources, often for a significant time period. It is invariably necessary to recognize that the benefits of developing one particular technology strategy alternative must be weighed against the costs of foregoing other opportunities.

There are a number of issues to be resolved through the evaluation efforts accomplished as part of the systems management of technology development and transfer. These include:

Each of these relates to the criteria that will actually be used for evaluating emerging technology development strategies. I have discussed many of these criteria in our efforts this far. Dutton and Crowe [12] provide an excellent summary of many appropriate evaluation attributes:

I have rephrased these as attributes for a multiple attribute evaluation effort. These attributes may be viewed in several ways. For example, they can also be seen as ingredients at the various gateways for the development of emerging engineering technologies shown in figure 3. In addition to obtaining an evaluation of proposed technologies and associated development and/or transfer strategies, an appropriate approach to evaluation should also allow for full exploration of the needed functions to insure satisfactory development of appropriate technologies. To do this, people are needed who fill the roles of [41]:

These needs should also be translated into attributes that can be used to measure the success of a particular technology and development strategy.

The assessment and evaluation of alternative technologies proposed for development may be approached through the application of formal decision theoretic methods. A major objective in technology assessment and evaluation is forecasting the potential costs and the resulting effectiveness of technology development or transfer. It is desirable to minimize the error associated with this prediction. If a set of inductive beliefs do not conform to those associated with the use of the probability calculus, then the expected error can always be reduced by modifying those belief values to conform to the calculus. Multi-attribute utility models are generally additive. They calculate utilities of an alternative or outcome by a weighted addition of the utility values of the alternative or outcome across the multiple attributes. The reasons that support using these MAUT models are that the necessary and sufficient conditions for these models to hold are well known. Further, the very important notion of value independence results. A cost effectiveness model [53, 58] for evaluating alternatives may be utilized.

The broad goals of cost-benefit analysis are to provide procedures for the estimation and evaluation of the benefits and costs associated with alternative courses of action. In many cases, it will not be possible or appropriate to obtain a completely economic evaluation of the benefits of proposed courses of action. In this case, the word "benefit" is replaced by the multi-attribute term "effectiveness."

Identification and quantification of the benefits and costs of possible alternative courses of action for technology development and/or transfer is a difficult task. It is generally not as difficult perhaps as formulation of the issue and identification of the alternatives themselves, but it is still not easy. Here I use the word benefits to mean the possible effects of a project. These include the totality of both positive and negative benefits, or disbenefits. We must first identify benefits, and then we should quantify them by assigning a value to them. Many benefits (and disbenefits) will be intangible and will occur to differing groups or individuals in differing amounts. Problems with intangibles may be especially difficult in the public sector, where agencies are designed primarily to deliver services or public goods rather than products for individual consumption. A major goal of a private sector organization is profit maximization, and it is relatively easier to measure profit as a benefit. Often there will be a variety of reasons why people will be uncomfortable with providing a strict economic measure for benefits. The word "effectiveness" is often used instead of benefit when a strictly economic valuation is not needed. When effectiveness is substituted for benefit we obtain a cost-effectiveness analysis. The benefits of a public service are much more difficult to define because they are intangible or indivisible (or both).

The political environment of many public-sector efforts further complicates the task, and variables other than those associated with efficiency economy, and equity should be measured. This suggests a multiple perspective approach to effectiveness. Among the many possible perspectives that need to be considered are economic, technical, legal, social, and political.

In cost-effectiveness analysis, we desire to rank projects in terms of economic costs, and in terms of effectiveness. The reason for this is that there are non-commensurate attributes of a project. Certainly we would wish to eliminate conspicuously inferior projects, that is to say projects that are more expensive and less effective than other projects, from consideration for selection. Beyond this, a cost-effectiveness analysis does not specify which of several projects is "best." This can be accomplished if one is willing to trade off cost for effectiveness, such as to obtain a "scaler performance index." It can be done by considering cost as one of the attributes in the effectiveness evaluation approach selected.

The effectiveness of an alternative is the degree to which that alternative is perceived as satisfying identified objectives. The effectiveness assessment approach described here provides an explicit procedure for the translation of quantitative evaluation of alternatives when the impacts of the alternatives are described by multiple attributes. This is accomplished by identifying and organizing the attributes of event outcomes (or alternatives, if there are no probabilistic uncertainties that influence the outcome that will result from alternative selection) into a tree-type hierarchy of attributes that is used together with measures of effectiveness to compare alternative technologies and development strategies as a basis for choice-making. Generally, an effectiveness assessment study involves a number of major analytical steps and is illustrated in figure 12. The final results of a cost-effectiveness assessment are used for comparison, ranking, and prioritization of the identified technology development alternatives according to effectiveness. This effectiveness assessment can be very useful for interpreting and evaluating the results of an analysis effort. In order to use the approach, we need a set of attribute of objective measures, information on the relative importance of attributes or objectives, and sufficient knowledge about project alternative scores and their outcomes to be able to assign effectiveness scores to the attribute measures that characterize the impacts of each outcome.