Performance Case Study:
Cellular Infrastructure

Product Description

Tandem Computer's Integrity Systems division (now part of Compaq) and Motorola's Land Mobile Product Sector collaborated to port the call processing infrastructure for the iDEN® product family to Tandem/Compaq's NonStop™ Integrity S-Series Servers (a product line of fault tolerant UNIX servers).

Performance Requirements

There were three primary performance requirements:

• Call setup time (defined as maximum latency for 99% of all calls)
• Maximum CPU utilization on server.
• Minimum number of handsets (subscribers) to be supported on each server while meeting the above requirements.

Performance Support Activities

Tandem assigned a performance analyst to work full-time on the joint project from its inception In addition to having clear and consistent performance goals, this project had the benefit of a variety of measurement data. Motorola had measurements of the previous generation of the call processing application running on a non-Tandem UNIX system. Tandem had measurements of overhead for fundamental system tasks, such a context switches and message processing by protocol stacks. With some adjustments, this data gave a rough approximation of how the new application would behave on the target platform. Rudimentary modeling based on the available data had a significant effect on the direction of the project.

The first challenges arose during negotiations to establish the joint development effort. Preliminary analysis indicated that the only way to meet all the performance requirements was to adopt an input/output driver scheme with close integration between the operating system and the application. This made it impossible to delineate resource consumption (such as CPU time) between application and operating system. The Motorola team was accustomed to requiring performance measurements that demonstrated that resources consumption for non-application tasks did not exceed a specified budget. The tight integration of application and system made such an acceptance test impossible. Instead, the two teams agreed to jointly make all reasonable efforts to meet the performance goals. Motorola also agreed to share closely held data about call patterns because it was essential to accurately predict system performance.

Preliminary analysis also showed that the fastest server hardware available when the application port was released would not meet performance requirements. As a result, Tandem accelerated development of some planned hardware performance improvements.

Once the preliminary analysis was done, the next challenge was how to more accurately model and predict the performance of the completed system. Analytic methods alone would neither deliver the required data nor give an acceptable level of confidence. Because of the novel nature of many software components, realistic performance measurements would not be feasible until late in the development cycle. This left simulation as the only way to get the needed performance data in time. Therefore, a detailed simulation was written with the aid of the SES (now Hyperformix) Workbench™ product. The arrival of new call requests was modeled probabilistically, while the internal messages and other processing steps for each call were modeled explicitly. Processing times were based on measurement data from other platforms, scaled to the new platform. The simulation also accounted for the overhead of context switches (changes in active UNIX process). As better measurement data became available, it was easily incorporated into the model by changing simulation parameters.

Towards the end of development, one developer ran an benchmark left over from previous versions of the application and stated his results showed that the application would not meet its performance requirements. This was a false conclusion based on an incomplete understanding of how drivers and application interacted on the new platform. Because the project's performance analyst had worked closely with key developers on both teams, had made accurate predictions previously, and had detailed simulation results available, he had the credibility to explain the situation and avert any "panic" about performance.

The delivered product meet all its performance requirements. The final simulation prediction for CPU utilization was within 2 percentage points of the measured value. During a project review, top management cited the key contribution of performance analysis to the successful outcome.

Observations

The experience of this project offers some important lessons:

• Note the synergy of multiple analysis techniques. The combination of performance measurement, analytic modeling, and simulation made best use of available information and provided design guidance when it was needed.
• Even simple analysis can avert disaster. Had the project gone ahead with a more conventional I/O architecture and without the faster hardware, it would have missed its performance requirements by a wide margin, requiring months of hardware and software rework. At the business level, there would have been widespread controversy and perhaps the loss of a large sale.
• The social aspects of a project can be as important as the technical ones. By working closely with the developers, the performance analyst gained detailed knowledge of the product that lead to more accurate modeling. He also established the credibility needed to defuse a "panic" about the driver benchmark late in the project.

Return to the case studies page.
Return to the FrontRunner Computer Performance Consulting home page.