Making sure your automated test suites give you confidence
16 September, 2020
by Nick
In this article, I want to present an idea around how to ensure your test automation gives you the confidence you need.
An automated test suite should ideally be reliable, maintainable and fast. When starting out writing automated tests it is easy to dig yourself into an automation black hole. Automating in an unstructured way can result in a bunch of tests that don’t give you the confidence you need to release quickly.
Quite often, a Test Engineer will be dropped into a feature team that is working on new functionality and they will diligently start writing tests around the changes that are being developed. The short term drive to provide immediate feedback and demonstrate the value of the automated tests can often lead to problems further down the line as the test suite starts to grow.
The problem with writing tests by “feature”
The term “feature” is often used to define a small piece of work that doesn’t fundamentally change the business processes that an application allows. These features might also correlate very closely with something like a Jira ticket. Structuring your tests according to the way your application organically evolves can lead to a rather unstructured set of tests.
Imagine a web application to purchase car insurance from a large insurance company that offers multiple products. The typical steps might be as follows:
A newly formed feature team is responsible for adding a new Breakdown Cover option to the Additional Options that are available when purchasing insurance.
The Test Engineer might be tempted to dive straight in and write tests that align with the feature team’s defined acceptance criteria e.g.
AC1
Given I am a new customer
And I am purchasing car insurance
When I have selected my level of cover
And I am selecting my additional options
Then I am given the option of adding Breakdown Cover
It’s a concise AC that is easy to automate as a test. Verify that Breakdown Cover is available as part of the Additional Options step.
AC2
Given I am a new customer
And I am purchasing car insurance
When I have added optional Breakdown Cover
And am viewing the summary of my cover and options
Then I am able to see that Breakdown Cover is listed
Another simple AC to automate. Just write another test to verify that Breakdown Cover is correctly displayed as part of the Quote / Summary.
AC3
Given I am a new customer
And I am purchasing car insurance
When I have purchased car insurance
And have selected the optional Breakdown Cover
And have completed payment
Then I am taken to the Confirmation of Cover page
And the optional Breakdown Cover is displayed under the additional options.
The final AC for this change requires that the purchase of Breakdown Cover is displayed as part of the Confirmation of Cover. Writing another test for this AC will ensure that all the ACs have an associated automated test.
Part of the problem with this approach is that each test will duplicate part of what the other tests are doing. It’s only three new tests, but what happens next week when the feature team adds Legal Cover and an option to reduce Policy Excess? This approach will lead to a sprawl of tests which test relatively low level aspects of the application’s UI and functionality. They are useful tests, but they might not be delivering the best value.
A better approach
The above scenario has been completely made up but it is a common pitfall that teams can fall into when building up a test suite feature by feature. Over time, you end up with a pile of tests that don’t adequately cover the application and as a result, don’t deliver the confidence the team requires to release changes quickly.
Conversely, through the sheer number of tests being written, you might get the confidence, but you also have a large maintenance burden and a long run time for the pack to complete. You might also start to find it is difficult to understand coverage.
An effective approach to address this is to take a step back and think about the processes the application will support (and need automating). In the case of our car insurance web application, we could model it as a flow diagram as below.
Instead of asking "What steps do I need to take to verify this AC?", try flipping the problem so you are answering "What business processes relate to this AC" and "What needs checking to verify it?"
In the feature example above, all of the acceptance criteria would relate to the process of purchasing a policy. I would argue that a single test that covers the end to end process would be sufficient. If you build that test, you can add additional assertions as appropriate to accommodate additional changes over time.
When you start thinking about the changes being made in terms of how they impact processes, and how you can verify the changes as part of that process, you will end up with a much more concise set of tests that will give you better confidence around end to end functionality.
If we paused for a moment and took the time to model our application in terms of the business flows, this would serve as a useful tool to guide our testing. It might even get us thinking about tests around a feature that might not be obvious from the original acceptance criteria.
In the diagram above, we have our green happy path. The red boxes represent alternative journeys that would be terminal end states for a user.
Understand your test coverage
Testing is a complex activity to get right. It might well be that you make a conscious decision to write tests for every AC. You probably want to do a mix of testing at different levels of the testing pyramid (you should!). The automated tests should also be supplemented with good manual testing around new features but it is important to not try and automate everything.
Being able to point to a diagram of your processes and explaining which parts of it have been automated is a great way of showing what your coverage is, and to help others understand where you might need to focus effort for upcoming changes.
Further Reading
In writing this article, my ideas have been formed both from experience, and from reading the writings of others. Listed in no particular order are other articles that I think are worth mentioning:
How to implement UI testing without shooting yourself in the foot