page 2  (39 pages)
to previous section1
3to next section

Analysis of Benchmark Characteristics and Benchmark

Performance Prediction

Rafael H. Saavedra

Alan Jay Smith


Standard benchmarking provides the run times for given programs on

given machines, but fails to provide insight as to why those results were

obtained (either in terms of machine or program characteristics), and fails

to provide run times for that program on some other machine, or some

other programs on that machine. We have developed a machine-

independent model of program execution to characterize both machine

performance and program execution. By merging these machine and pro-

gram characterizations, we can estimate execution time for arbitrary

machine/program combinations. Our technique allows us to identify those

operations, either on the machine or in the programs, which dominate the

benchmark results. This information helps designers in improving the

performance of future machines, and users in tuning their applications to

better utilize the performance of existing machines.

Here we apply our methodology to characterize benchmarks and

predict their execution times. We present extensive run-time statistics for

a large set of benchmarks including the SPEC and Perfect Club suites.

We show how these statistics can be used to identify important shortcom-

ings in the programs. In addition, we give execution time estimates for a

large sample of programs and machines and compare these against bench-

mark results. Finally, we develop a metric for program similarity that

makes it possible to classify benchmarks with respect to a large set of



? The material presented here is based on research supported principally by NASA under grant NCC2-550, and also in part

by the National Science Foundation under grants MIP-8713274, MIP-9116578 and CCR-9117028, by the State of Califor-

nia under the MICRO program, and by the International Business Machines Corporation, Philips Laboratories/Signetics,

Apple Computer Corporation, Intel Corporation, Mitsubishi Electric, Sun Microsystems, and Digital Equipment Corpora-


? This paper is available as Computer Science Technical Report USC-CS-92-524, University of Southern California, and

Computer Science Technical Report UCB/CSD 92/715, UC Berkeley.

? Computer Science Department, Henry Salvatori Computer Science Center, University of Southern California, Los

Angeles, California 90089-0781 (e-mail:

?? Computer Science Division, EECS Department, University of California, Berkeley, California 94720.