Application Software Testing: A Practical Guide to Improving Software Quality

APPLICATION TESTING: A PRACTICAL GUIDE TO IMPROVING SOFTWARE QUALITY

(If you're just starting to explore ways to improve your IT organization's software management practices, you may want to read the Software Development Survival Guide - Five Steps from Chaos to Control whitepaper. See "Where to Learn More" at the end of this whitepaper.)

QUALITY ASSURANCE

Software testing is a technique you use to measure, and thereby improve, software quality. Software testing fits in the broader category of software management practices known as quality assurance (QA), along with other practices, such as defect tracking and design and code inspections. The overall goal of QA is to deliver software that minimizes defects and meets specified levels of function, reliability, and performance.

The single most important QA practice you can follow is to record all identified software defects and other types of problems. Tracking problems is key to improving software quality, but discovering problems only after an application is put in production is disruptive and expensive.

Effective testing before production deployment achieves two major benefits:

- Discovering defects before an application is deployed allows you to fix them before they impact business operations. This reduces business disruptions from software failure or errors and reduces the cost of fixing the defects.

- You can estimate the extent of remaining, undiscovered defects in software and use such estimates to decide when the software meets reliability criteria for production deployment.

- Test results help you identify strengths and deficiencies in your development processes and make process improvements that improve delivered software.

Considering the first benefit, your own experience probably confirms what software researchers have discovered: The later a "bug" or other defect is found, the more troublesome and expensive it is. Defects in production software can severely disrupt business operations by causing downtime, customer complaints, or errors.

Software researchers have found that defects discovered in production can be as much as one hundred times more expensive to fix than those found early in the design and implementation process. Excessive numbers of defects also disrupt the software development process, because the development group "thrashes" as they try to manage the cycle of fixing defects in production code while adding new features or making other planned changes required by the business.

Software testing is just one way to discover defects earlier in the development process. Various studies indicate that well-conducted testing by itself may identify somewhere around 50 or 60 percent of the defects present in an application. Design and code inspections are another technique for discovering defects; and, interestingly, research indicates inspections can achieve error detection rates as high as 90 percent.

Whether or not your organization considers inspections, testing is still beneficial. For one thing, testing and inspections find different types of errors, so neither technique is sufficient by itself. But the main issue for most organizations is that inspections require substantial training and intensive team effort and are consequently a much bigger undertaking than improving your testing processes.

The second major benefit of testing (which applies to code inspections, as well) is that you gain some measure of how "buggy" a piece of software is before you decide to deploy it. For example, if your experience with testing reveals that, on average, about as many defects are found after deployment as are found during testing, then you can project a similar relationship for future projects.

Many programmers find it hard to believe that even thorough testing may still miss half the defects that are eventually discovered. But, as we shall see, it's impossible to test any nontrivial application completely. This is why, as helpful as testing is in finding defects before software is deployed, you still shouldn't view testing as a foolproof "filter" that can remove nearly all undesirable elements from application code, the way an air filter cleans impurities from the air.

If your software testing development process is producing a high rate of defects, you need to work on some part of your development practices to lower those rates, not just increase the amount of time spent testing. Testing gives you the data necessary to know how effective your other design and coding practices are at achieving software quality targets. Without consistent testing, not only will your production software be less accurate and reliable, but you also won't have a solid basis for deciding which changes to make in your development process.