Devops relies upon continuous integration practices to facilitate speed and quality. CI calls for developers to frequently integrate code into a shared repository, thereby initiating automated builds. This allows teams to detect and fix problems early. CI also creates the need for continuous testing. There are several definitions, but I define continuous testing as “all test activities performed while operating in a continuous integration environment.” Without continuous integration, you really can’t even begin to talk about how continuous testing plays a role in a devops environment.
Continuous testing isn’t easy to do, yet it delivers distinct advantages for development teams:
- Testing activities integrate fully into the development pipeline.
- It necessitates automated test coverage, which drives efficiency and standardization.
- Increased automation gives testers more time to use exploratory means to uncover hidden issues.
- Teams feel more confident about the quality of software before deployment.
While continuous testing can be highly efficient, there are pitfalls that can make it less so, such as automating tests that shouldn’t be automated and not storing tests as code with a version control system (VCS). While the first issue has more to do with test strategy, the latter makes it hard to maintain tests, reuse tests, and research issues.
Let’s dig into a few best practices to ensure success with continuous testing.
1. Ensure you have a complete foundation for test automation in place
A solid CI platform is critical. The G2 Crowd Grid for CI qualifies solutions based on this criteria:
- Developers can write unit tests that automatically execute.
- The system shows which tests have passed and failed;
- The system performs all actions to create a fully functioning build of the software after tests have passed.
Most teams will strive to automate unit, functional, and regression tests. Test automation software applications handle the execution of automated tests and reporting. Teams will also need tools to handle planning (i.e., requirements, user stories), version control for source code, and test management that integrates tests and testing activity into the devops stack for traceability and management. Users should be able to correlate tests directly to the source code and planning systems to improve collaboration between developers and testers.
2. Consider using BDD (behavior-driven development)
Continuous testing is an automation-first mentality, which means it aligns nicely with behavior-driven development. BDD isn’t the only way to accomplish continuous testing, but it encourages continuous collaboration among product owners, developers, and testers. Scenarios focus on how the future should behave for the user, enforcing clarity on development goals and better alignment between developers and testers. Features are usually stored with code, which makes integrating with CI systems easier. Finally, it’s easier to turn user scenarios into automated tests by using the Gherkin syntax, which is a staple feature for tools like Cucumber.
3. Empower QA teams
QA may create, edit, and run most of the test activities, yet in devops, QA’s influence is more than that. QA can ensure that each team member carries forth with continuous testing duties throughout the life cycle. If developers are not automating unit-level tests, QA has the right to speak up for the team. It can also enforce process, such as enforcing that automation engineers and test analysts store their tests with code. QA leaders can guide their teams on testing strategy, which includes determining the right mix of manual and automated testing as well as when to strike a balance between different types of testing. Finally, QA knows that a team can’t test every single scenario and will use its best judgment to say when it’s time to move on to deployment.
4. Be analytical about automated testing
It’s easy to say to automate everything, but that will be wasteful and inappropriate. There are distinct scenarios when manual testing is a superior approach. A rule of thumb is to automate all unit-level testing. It’s more complicated to write automated tests for functional, performance, and other forms of testing, so develop a process of elimination when deciding what to automate. For example, if 90 percent of your users have the same type of user profile, you might want to automate a test for logging in with that type of profile because any issues would affect 90 percent of your users. The risk of failed login for the remaining 10 percent isn’t high enough to warrant an automated test.
Take advantage of manual testers and their subject matter expertise about the application and user base. They can work closely with automation engineers to help identify the best areas for automated testing. Exploratory testing is also important for automation strategy, as testers gather new information about scenarios and potential issues. This intelligence can inform where it makes sense to automate testing.
5. Continuous testing goes beyond test automation
Automation is integral to making the continuous testing process work, yet it doesn’t apply only to running tests. You need automated processes that inform testers when there is a code change in the application. You need automated notifications when CI is complete and the next state is ready for test. You need automated provisioning of test environments to rapidly run your tests. Extending automation to other areas of the QA function reduces errors, keeps the wheels moving and prevents burnout.
Final words
There’s been much chatter concerning the cultural shift needed for QA teams as they transition to continuous testing. Leadership should understand and address those challenges and make pains to keep everyone on the same page. Meanwhile, a detailed strategy, with ample thought around tools, roles, and processes to support continuous testing helps organize and motivate team members. Without structure, QA staff will be sailing without a rudder and jeopardizing your larger agile and devops efforts.
This article is published as part of the IDG Contributor Network. Want to Join?
