Performance

Engineering
Services

Overview

Over the years, we have observed many organizations that have tried to adopt SPE. All of these organizations were of the opinion that SPE was beneficial and would help them overcome their problems with software performance. Some were successful; others were not.

We have also seen SPE applied in cycles. That is, after successfully applying SPE to new development there were no performance problems. When it was time for the next major release, managers thought Why do we have performance engineers, we don’t have any performance problems. They omitted SPE, had performance problems, re-established SPE, and so on, and so on.

There are lots of mistakes that can lead to failure. In our experience, many of them occur over and over — like antipatterns. From that experience, we have compiled our candidates for the top ten ways to ensure an unsuccessful SPE initiative. Clearly, this is a tongue-in-cheek list — no one wants an unsuccessful SPE initiative. So, each item in the list begins with a description of a way in which we have seen an SPE initiative fail. Then we discuss the consequences and a solution that will help avoid that pitfall.

Each month, we will present a different item until we have covered the entire list. Here is the latest installment.

Construction Project Analogy

Many of the Top 10 items make use of an analogy to the Grand Canyon skywalk project completed in 2007. This cantilevered walkway allows visitors to look through a glass floor to the bottom of the Grand Canyon 4,000 feet below. It is designed to hold 72 tons, survive a magnitude 8.0 earthquake 50 miles away, and withstand winds in excess of 100 mph. There are many ways this project could have failed to meet its requirements (e.g., utility, strength, stability, durability, architectural beauty, etc.). The analogies illustrate the point by drawing parallels between ways in which this construction project could have failed and the ways in which an SPE project could fail.

Focus on the Small Things

The analogy between "Focus on the Small Things" for SPE and the skywalk construction project is to focus on the details of the floor of the walkway, perhaps concentrating on optimizing the properties of the clear see-through material, while neglecting the overall architecture or structure of the walkway. The consequence is that the floor of the walkway might be beautiful, but if the overall structure is unstable then the walkway, and thus the floor, are unusable. There are two principal ways of focusing on the small things at the expense of your SPE initiative. This first is known as sub-optimization—devoting excessive effort to optimizing an area of the software that does not have a significant impact on performance. This issue has been recognized for many years [Fairley, 1985]. It arises because developers do not have useful information on which parts of the application have the most impact on performance or how a particular "optimization" in one area will affect global performance. In an extreme case, a local "optimization" can actually be a global de-optimization.

The second way of focusing on the small things is to focus on tuning as a way of managing system performance. This may sound strange since SPE is the antidote to the build/test/tune tarpit. However, it is easy to fall back into old habits—especially with your first SPE efforts. We saw one case where an organization rolled out an ambitious SPE plan but actually ended up with just another "test and tune" effort.

We have also seen instances where people have used a "test and tune" approach in which the software was unit tested after each increment of development. In these cases the individual units may have passed their test but, when the system was integrated, the overall result fell short of meeting performance requirements. Unit tests are typically run against a small subset of data and without multiple users. So the unit test performance is not representative of the production environment. It is also difficult to define realistic performance requirements at the unit test level without SPE performance models of the overall system.

Focusing on the small things is a forest-and-trees issue. Focusing on the small things often means missing opportunities for building-in performance as well as scalability. Performance can be orders of magnitude better when it is planned and designed into the system than when it is tuned into a substandard design.

Consequences

The primary consequence of sub-optimization is that you waste time and effort optimizing things that have little or no impact on the overall performance of the system. In the case of a local optimization that turns out to be a global de-optimization, you may waste a lot of additional time tracking down the source of the problem.

The consequence of focusing on tuning for managing software performance is that the likelihood that the application will not meet its performance requirements is increased. Waiting until the end of the project to tune the code implicitly assumes that the code can be tuned sufficiently to meet performance requirements. However, since most performance problems are introduced in the architecture or design phase of the project. Relying on tuning means that these problems are not detected until much later—when they are much more costly and time consuming to fix. If the application can be tuned, the time required for tuning will most likely cause schedule and cost overruns. If the architecture will not support the performance requirements, no amount of tuning will salvage the application.

Solution

SPE provides a way for directly addressing the issue of sub-optimization. Building and evaluating software performance models will tell you exactly which parts of the software are most used for each of your key performance scenarios. You can use this information to design the software (not optimize the code!) so that resource usage is reduced and performance requirements are met. Solving the system model will allow you to evaluate proposed local optimizations to ensure that they are not global de-optimizations.

Similarly, following the SPE process will help you avoid relying on tuning to meet performance requirements. Early in development it is possible to construct models based on the proposed architecture that will allow you to evaluate whether that architecture will support your performance requirements. If not, changes can be made before proceeding. Using the models, along with measurements, to track performance of the evolving software will alert you to potential problems and make it possible to address them when they are easier and less costly to fix.

Everybody has a tuning "success" story in which some "caped hero or heroine" saved the day by tuning the application so that it performed well enough to ship (OK, so it was a couple of months late). We don't want to detract from the dedication and skill of these heroes and heroines but these "successes," when viewed in an organizational context, are in reality failures. They represent a failure of the development process to properly manage performance and they cost the organization large sums of money in tuning costs, lost revenue, and opportunity costs. There are also intangible costs such as damaged customer relations and loss of reputation. Tracking these costs and quantifying the costs of tuning will help bolster your argument to do things right.

References

[Fairley, 1985] R. Fairley, Software Engineering Concepts, New York, McGraw-Hill, 1985.

Click to View Previous Installment