Automated Tests Simplicity

This article originally appeared, in Dutch, in the TestNet news magazine in December 2019.

Does the following scenario ring a bell?

‘It all started out so well when we decided to adopt test automation to support our testing activities… We invested time to do a thorough tool selection, and we carefully selected tests and product areas that we wanted to cover by automation. The first couple of months, everything went pretty smoothly, and everybody was happy with the results.

At some point in time, however, things started to slide… Our suite of automated tests grew, maintenance started to consume more and more time, and some tests started to fail for no apparent reason. Slowly but surely, the initial euphoria started to wane, and people started to lose confidence in the results and the information that the automated tests produced. Currently, I’m the only person that invests any time at all trying to maintain and run what’s left of our test automation…’

In the 14 years that I’ve been working in the test automation field, I’ve seen and worked with quite a few organizations and projects where a version of this scenario occurred at some point in time. Sometimes, I was the one that was hired to troubleshoot, often I was part of the problem. What started out as a small, but valuable set of automated tests over time grew into an enormous test code repository that required

• a lot of time to execute,
• even more time to maintain, and
• an enormous amount of time to explain and hand over to others (not something I’m proud of being a contractor, in hindsight…)

But what causes test automation efforts that start out on a positive note, providing tangible value, to slowly deteriorate into an inextricable mess of libraries, test suites, scripts and code (or low code artifacts)?

The answer to this question is simple.

Or, more precisely: a lack of simple.

I see a lot of teams and organizations (and again, I’ve been as guilty of this as the next person) spending a lot of time creating new tests, expanding the features of their test automation framework in an effort to -hopefully- increase their test coverage.

What I see far fewer of these teams and organizations spend enough time on is simplifying, deduplicating and cleaning out their test suites.

To some extent, that’s understandable and even completely human. It’s hard to say goodbye to things we have spent a lot of time and effort, and therefore money, to create in the first place. There’s even an official term for this: the sunk cost fallacy, defined as ‘the continuation of a behavior or endeavor as a result of previously invested resources’.

But that doesn’t mean that reconsidering your current tests and maybe even throwing some of them away is a bad idea. Quite the contrary, I think it’s a great idea, and I’d love to see people spend a little more time on the following activities:

• Code reviews for test code. What seems like a valuable, understandable and maintainable tests to person A can look a lot less useful to person B, especially when person A is the one who created the test. Code reviews are a great way of uncovering these differences in perception, providing valuable input to improve the quality of your tests. And please don’t think you’re exempt from code reviews just because you use a low code automation solution. Because the initial barrier of entry to these tools is generally lower, it’s maybe even more important to keep a close eye on the quality of the tests created therein.
• Test your tests. As the adage goes: “Never trust a test you haven’t seen fail!”. Is the test you’ve built capable of failing? And when it does fail, does it fail for a clear and unambiguous reason? Techniques like mutation testing can help you increase the quality of your tests, at least at the unit testing level.
• Cleaning out your test suites. If developers regularly take time to refactor their application code (they do, right? Right?), why wouldn’t test automation engineers do the same for their automated tests? That test you’ve built a year ago, is it still valuable? And what about its reliability? Again, don’t be afraid to throw away tests that exceeded their ‘best before’ date.

The underlying principle with all these activities?

“Keep it simple”.

Simple automated tests are well-structured, readable, preferably check exactly one thing and only fail for the right reason: a potential defect in the application under test. Simple tests are also easier to maintain, and sharing knowledge about simple tests is easier and less time-consuming.

Sounds wonderful, right? There’s just one thing that you need to know:

Simple is not easy.

Creating complex tests is relatively easy: adopt a tool, look at the features of that tool, think about how you can use those features to create tests, and a couple of visits to Google and StackOverflow later, you’ve got yourself a test that produces the desired result. Or at least one that shows one of those appealing green check marks.

That’s all well and good… Until you are asked to maintain that test after a change in the application under test, or when you want to reuse parts of that test for other tests, or when you want to make that test part of a Continuous Integration pipeline, running the test against a different environment and configuration.

Simple automated tests, on the other hand, are the result of thinking hard about, among others things:

• What should and should not be automated
• What components and layers of the application play a role in obtaining the information that the test should produce
• What would be a good design of the test and the underlying automation solution (the framework)
• What would be a good approach with regards to test data and test environment management

Simple, elegant tests are often the result of hard work, continuous improvement and throwing away what doesn’t work. And yes, that takes time, effort and as a result, money, and it does require specific and in depth knowledge. I am confident, however, that if you want to reap the benefits of your test automation efforts in the long run, you have no other option than to continuously strive for simple (elegant) tests.

Or, as the Dutch software engineer and researcher Edsger Dijkstra already said way back in the 80's:

“Elegance is not a dispensable luxury but a quality that decides between success and failure”

How simple are your automated tests?

This article originally appeared, in Dutch, in the TestNet news magazine in December 2019.