Analysis of Benchmark Characteristics and Benchmark
Rafael H. Saavedra
Alan Jay Smith
Standard benchmarking provides the run times for given programs on
given machines, but fails to provide insight as to why those results were
obtained (either in terms of machine or program characteristics), and fails
to provide run times for that program on some other machine, or some
other programs on that machine. We have developed a machine-
independent model of program execution to characterize both machine
performance and program execution. By merging these machine and pro-
gram characterizations, we can estimate execution time for arbitrary
machine/program combinations. Our technique allows us to identify those
operations, either on the machine or in the programs, which dominate the
benchmark results. This information helps designers in improving the
performance of future machines, and users in tuning their applications to
better utilize the performance of existing machines.
Here we apply our methodology to characterize benchmarks and
predict their execution times. We present extensive run-time statistics for
a large set of benchmarks including the SPEC and Perfect Club suites.
We show how these statistics can be used to identify important shortcom-
ings in the programs. In addition, we give execution time estimates for a
large sample of programs and machines and compare these against bench-
mark results. Finally, we develop a metric for program similarity that
makes it possible to classify benchmarks with respect to a large set of
? The material presented here is based on research supported principally by NASA under grant NCC2-550, and also in part
by the National Science Foundation under grants MIP-8713274, MIP-9116578 and CCR-9117028, by the State of Califor-
nia under the MICRO program, and by the International Business Machines Corporation, Philips Laboratories/Signetics,
Apple Computer Corporation, Intel Corporation, Mitsubishi Electric, Sun Microsystems, and Digital Equipment Corpora-
? This paper is available as Computer Science Technical Report USC-CS-92-524, University of Southern California, and
Computer Science Technical Report UCB/CSD 92/715, UC Berkeley.
? Computer Science Department, Henry Salvatori Computer Science Center, University of Southern California, Los
Angeles, California 90089-0781 (e-mail: firstname.lastname@example.org).
?? Computer Science Division, EECS Department, University of California, Berkeley, California 94720.