Performance Analysis Techniques

Overview

Techniques for computer performance analysis are divided into three broad classes: analytic modeling, simulation modeling, and performance measurement (benchmarking). All analytic and simulation techniques require the construction of a model: an abstract representation of the real system. An analytic performance model is a mathematical construct, while a simulation model is a specialized computer program. Much of the art in performance analysis lies in selecting a good model -- one that captures the salient aspects of the system without obscuring them in a mass of irrelevant details.

The third technique, performance measurement (including the use of benchmarks) does not use models but instead relies on direct observation of the system of interest, or a similar system.

No one technique is best in all cases, and most projects are best served by using multiple analysis techniques.

Comparing Techniques

A number of factors must be considered in selecting the appropriate performance analysis technique for a project. Some of the key questions are:

• When will a working prototype be available? Analytic and simulation models can be used at any time, but a working implementation of the product is needed to do measurements. Furthermore, the implementation must be robust and near its final level of optimization to provide relevant performance measurements.
• How much time is available for analysis? Analytic models can be constructed quickly and with limited information, while simulation and measurements require substantially more effort to set up. In the absence of any existing infrastructure for performance measurement, a simulation typically takes less time than a performance measurement.
• What are the analyst's skills? No performance analysis technique is foolproof, but constructing analytic models requires some practice and knowledge of specialized mathematics. Simulation and measurement may be more accessible to developers and other non-specialists who need to do performance analysis.
• What tools are required? In many cases, analytic models can be constructed with general purpose software such as spreadsheets. While simulations can be written in a general purpose programming language, in almost all cases it's best to use specialized simulation software. Performance measurements need hardware/software to generate the load and collect data.
• What level of accuracy is needed? Analytic models contain many simplifications and assumptions and so yield only approximate answers. While this is sufficient in many cases, there are situations where greater accuracy is needed. Simulation can offer greater accuracy at the expense of more coding effort and run time. Measurement are the most accurate of all provided the instrumentation is adequate.
• Are there tradeoffs to consider? Analytic models provide a quick way to compare different product configurations. Simulations can also evaluate tradeoffs, but each alternative requires additional effort to code and validate the simulation. Measurements are the least flexible: if configuration can be changed at all, it's usually a laborious effort.
• What metrics are needed? Analytic techniques provide the basic metrics: throughput, latency and utilization. Simulations can be programmed to collect any metric required by a study. The set of metrics available from measurement is determined by the instrumentation installed. Also, the overhead of excessive data collection can distort measurement results.
• Does the uncertainty of results need to be quantified? Analytic models provide only the mean (average) value of performance metrics. Simulations can provide information about variability as well, especially when the same configuration is simulated multiple times with different random number seeds. The best information about variability is provided by making multiple measurements.
• What resources (time, money) are available? Analytic models generally require the smallest investment in time and money. In most cases, measurement is the most expensive option, while simulation is medium cost option. There are exceptions: measuring a configuration of a few PC's may be cheaper than buying a commercial simulation package. And in many cases, performance measurement can leverage resources used for other purposes, such as functional testing.
• How "salable" are the results? In many cases, the results of a performance study must be "sold" to a skeptical audience. Performance measurements are generally perceived as the most credible, while modeling results typically require that the audience either have previous experience with performance modeling or trust in the analyst's ability. Note that any analysis method can yield erroneous and misleading results in the hands of the careless.

The following table summarizes these considerations:

In many instances the best answers come from a hybrid approach. Simple analytic techniques can be used to extrapolate from existing measurement to new systems and applications. Simulation and benchmark results can provide inputs to analytic models. And it's always a good idea to validate the results obtained with one method by a second independent analysis using another technique. See the case studies page for examples of the interplay between analysis techniques during real projects.

Acknowledgment: Much of this discussion is based on section 3.1 of The Art of Computer Systems Performance Analysis by Raj Jain.

Learn more about analytic performance modeling.
Learn more about simulation performance modeling.
Learn more about performance measurement.
Return to Understanding Computer Performance Analysis page.
Return to the FrontRunner Computer Performance Consulting home page.