Step 5:
Select Analysis Techniques

Techniques for computer performance analysis are divided into three broad classes: analytic modeling, simulation modeling, and performance measurement (benchmarking). All analytic and simulation techniques require the construction of a model: an abstract representation of the real system. An analytic performance model is a mathematical construct, while a simulation model is a specialized computer program. Much of the art in performance analysis lies in selecting a good model -- one that captures the salient aspects of the system without obscuring them in a mass of irrelevant details.

The third technique, performance measurement, does not use models but instead relys on direct observation of the system of interest, or a similar system. Benchmarks are a special type of performance measurement. Typical benchmarks define both a workload and set of performance metrics. In most cases, benchmarks are published by independent organizations and provide a way to compare the performance of products from multiple vendors.

No one technique is best in all cases, and most projects are best served by using multiple analysis techniques. Analytic techniques can quickly supply approximate answers at the start of product development. Even a simple analysis is often very useful for identifying bottlenecks or areas requiring closer investigation. As development proceeds and more detailed performance information is required, simulation, measurements or a combination are used. Performance measurement is a good choice when the new product is similar to an existing one and most of the infrastructure required for the measurement already exists. On the other hand, simulation is often the only way to evaluate the performance of a novel design.

In many instances the best answers come from a hybrid approach. Simple analytic techniques can be used to extrapolate from existing measurement to new systems and applications. Simulation and measurement results can provide inputs to analytic models. And it's always a good idea to validate the results obtained with one method by a second independent analysis using another technique. For examples of how multiple analysis techniques are used in real projects, please see the case studies page.