“Failing fast” lies at the heart of agile. Teams want continuous integration (CI) to provide feedback on their latest updates as soon as possible. CI test results are the primary barometer that developers use to determine whether it’s safe to move on to the next development task, or if they inadvertently broke functionality that users have grown to rely on.
With more extensive and more effective regression testing during CI, you’re much more likely to spot critical problems as soon as they’re introduced—which is when they’re fastest, easiest and cheapest to fix. However, given the frequency of builds in most agile processes, there’s simply not much time available for test execution. Developers expect feedback in a matter of minutes, but most regression test suites—especially in Global 2000 organizations—take hours (or days!) to execute. This seems to force a trade-off: Settle for sluggish CI or scale down testing.
Having slow CI impacts productivity across all the team members waiting to review, extend, document, and test the newly integrated functionality. The closer the end-of-sprint deadline, the more painful each minute of waiting seems. Of course, you can (and should) accelerate the process with parallel/distributed test execution and similar technologies. But ultimately, if you want to accelerate the execution of a large enterprise test suite, you will need to make some hard decisions about what tests do not need to be executed for each build.
Ways to speed up (and scale down) testing
One common approach to speeding up regression testing is to slash end-to-end tests. This is a huge mistake. As I’ve noted in past columns, I know that unit tests are valuable for exposing implementation errors. However, I just don’t see unit tests catching the issues that really impact the end user, which are largely integration issues. If you skip end-to-end tests designed by a professional tester who really understands the domain, you’ll inevitably miss a lot of issues that have a very direct impact on the user experience.
Risk-based testing is another “test acceleration” strategy, one that has a solid track record of delivering impressive results. But many organizations shy away from this approach because they fear it requires too much overhead. It doesn’t. I’ll explore this in detail in one of my next columns, but long story short: Unless you happen to have unlimited time and resources, you must use some level of risk-based prioritization to guide both your test case design and test execution efforts. Failure to do so places your business at too great of a risk.
Test impact analysis
For this column, let’s focus on an approach that’s starting to gain attention in the agile world: test impact analysis. This technique, as pioneered by Technical University Munich spinoff CQSE, rapidly exposes issues in code added or modified since the previous test run by applying two main principles:
- Correlate all regression tests (even end-to-end regression tests) to code and select only the tests associated with the latest round of code changes. Why waste time executing tests that have no chance of uncovering defects in your new or modified code?
- Order those regression tests based on their likelihood of detecting a problem—and prioritize execution of the ones that are most likely to expose defects. If your builds are set to fail upon the first reported test case failure, you might as well reach that point as soon as possible.
This enables you to find the lion’s share of defects in a fraction of the time. In fact, studies show that this approach uncovers 80 percent of faulty builds in 1 percent of the complete test execution time, and it uncovers 90 percent of faulty builds in 2 percent of the complete test execution time. (This study, performed by CQSE, analyzed both proprietary and open source code, and covered software written for business information systems as well as embedded systems.) In other words, you can speed up testing 100 times and still find most problems. It’s ideal for optimizing continuous testing.
Obviously, this new way to “fail fast” results in much tighter feedback loops—which means that failing builds get resolved sooner and working builds are delivered faster. It also accelerates agile team velocity in a few other ways:
- Test < > code correlation makes it easier to determine where additional tests need to be added—and when sufficient code coverage has already been achieved.
- Test < > code correlation also streamlines the defect remediation process. Especially with end-to-end tests, finding the code responsible for a test failure can be like searching for a needle in a haystack. With this correlation, you get a very precise path from the test sequence detecting a problem to the code that’s responsible for it.
- By squeezing more—and more effective—testing into a short period of time, you reduce the number of defects reported late in the process and after delivery. This ultimately results in less time wasted on bug fixes and rework, fewer midsprint interruptions, and more resources to focus on innovation.
Learning more about test impact analysis
Test impact analysis can be applied to any type of test—including unit tests (such as JUnit) and automated functional tests (Selenium, my company’s Tricentis Tosca, etc.) as well as manual tests. With test impact analysis, a specialized set of test cases is selected and executed for each test run:
- Static analysis identifies which code has been modified or added since the previous test run. That new or modified code is visualized for easy coverage tracking.
- Dynamic analysis correlates the regression test suite (which, ideally, was designed to focus on the top business risks) with the new or modified code in this build.
- Regression tests that do not touch new or modified code are eliminated for this test run.
- Additional analysis determines which tests are most likely to expose a defect in the new or modified code.
- The remaining tests are sorted so that those most likely to expose a defect are executed first.
- Tests are executed in the optimal order, then merged correlated code coverage from all the executed tests (unit tests, end-to-end tests, etc.) is visualized versus the new or modified code map—making it simple to spot coverage gaps.
This article is published as part of the IDG Contributor Network. Want to Join?
