Transform your test automation to actually work
This article was co-authored with Eran Kinsbruner, a test industry veteran and thought leader.
Despite advances in test automation, low/no code and others, testing remains blocker for software innovation and development velocity. Why? Because while there is (and was) a tremendous investment in developer [tools|process|people], the testing industry has not changed much, not really. Test professionals (and developers, to some extent) are still busy creating scripts, maintaining scripts, managing tests and test data.
To make things worse, requirements for test have also evolved: functional testing is no longer sufficient. To win users’ hearts, you need a functional, visually pleasing, performant, secure product.
Clearly, everyone, especially developers have keen interest in learning about defects as early as possible.
Last on the pile of issues, is the topic of script maintenance. The application code changes, scripts that were valid a minute ago, have become flaky. To give an idea, a typical nightly test suite has a few thousands of test executions. You can imagine some flakiness in the login function. It is estimated that up to 70% of a test professional day be spent on script maintenance.
In my years in the testing space, I’ve seen decision makers frustration. Investment is made, no doubt, with the expectation of high bar of application quality, and locked in step with the pace of developers. But that’s not the case. The orchestration of (regression test scripts X platforms X schedule X test data) +new test for code written in sprint, that’s just an overwhelming task to manage.
Decision makers have made the investment in testing to FIND DEFECTS BEFORE THEIR USERS DO. Instead, their teams are building test automation. It’s not the same.
Can this change? I argue-yes.
Let’s set some objectives:
1- We’re looking to get rid of authoring and maintaining scripts for existing functionality (regression)
2- We’re looking to get rid of test management (bucketing tests into suites) and schedules
3- We’re looking to get rid of test data management
4- We’re looking to get rid of the existing platforms to be used in the regression tests
Ideally…
5- We’re looking to ease the authoring of scripts for NEW code
6- We’re looking for smart/efficient (more on that later) test management
7- Could we think about the test script as essentially a navigation mechanism through pages, which can yield into visual, accessibility, performance and other tests per page (or mobile view)?
8- Can we use this system to identify “sensitive” areas of the code, for the benefit of coders, as well as accelerated testing?
DISCLAIMER: Let’s agree on one thing- The above does NOT apply to all of your test cases. There are going to be test cases that are difficult to automate and frankly, maybe not worth it. But what if the above could work for 80% of your testing?
I say yes. Modern technology can make the above happen. We have the means to do two things:
1- Take the majority of test automation practice AWAY from the users.
2- Provide actionable insights: provide the defects to developers, provide analytics to test and R&D leads to better enable their teams to write better software.
HOW?
Phase 1: Let’s build a set of regression tests, and maintain them.
Hypothesis: Can product data be used to produce the most common user flows, platforms (from analytics tools), and non-PII data for testing? Yes.
This is indeed far easier said than done, but totally doable. For example, test data can be analyzed for the PII elements, to be replaced by mocked data that complies with the necessary format (RegEx).
What about maintenance? Well, as it turns, the low/no code industry was able to prove that most of the necessary changes in existing scripts relate to manipulation of the object locators, not necessarily a rebuild of the entire script logic. Many of those tools essentially record all of the object locators (ID, CSS, Name etc.) and try to alternate if one fails, eliminating a significant amount of false failures and self-correcting.
Phase 2: Let’s build a set of tests to reflect the in-sprint code creation.
Hypothesis: English-based feature requirements can be translated into test scripts.
In reality, this has been supported for a long time, in the form of Cucumber.
Let’s complete the picture now:
So now we have a set of scripts auto-authored based on requirements, running on a frequent schedule. What about platforms? Well, typically early tests run only on one platform for efficiency reasons (for example, web: Chrome/windows).
Does this approach require hand-holding? absolutely. When there’s a new mobile device in the market, your users will not have already used it, so it will not show in the analytics. It needs to be inserted into the rotation manually. The point is, though, the above can work reliably for the majority of your testing.
We covered our 1–5 objectives about. Let’s talk about the rest of them.
6- We’re looking for smart/efficient test management.
- Can this system detect flaky tests and move them out of rotation, open a Jira ticket and keep them out until they are marked fixed?- Yes.
- Can this system detect overlap in code coverage and eliminate redundancy? Yes.
- Can this system designate a set of critical tests that need to pass before a large regression test suite runs? Yes.
- Can this system automatically migrate a set of tests from time-critical test suites (such as smoke test suite) to regression? Yes.
7- Could we think about the test script as essentially a navigation mechanism through pages, which can yield into visual, accessibility, performance and other tests per page (or mobile view)?
We definitely can. Consider a script as a navigation path, traversing the user experience. On each page, run a visual test, performance, Google lighthouse set of tests, accessibility etc. Here’s another example: With modern visual testing, you can compare the entire DOM tree of a page, essentially a “full page functional regression”. Why not do that? For the same script, get your value multi fold.
There is one caveat, which is that some of those tests take time (lighthouse as an example). Don’t run all of them on every page repeatedly, certainly not when you’re pressed on time.
8- Can we use this system to identify “sensitive” areas of the code, for the benefit of coders, as well as accelerated testing? Yes.
This one is actually very interesting to me. Is there an intelligent way to tell a code is more “impactful” to users and the business? I argue- yes. What happens when code is more “impactful”? some elements are subjective: user annoyance, bad reviews etc. But others are easily measurable: Lost carts (in retail), outages, as example. Another way to measure is how quickly a developer reacts, and by “react” I mean commit a fix to production. Over time, one can develop statistics to show that the checkout code is much more sensitive to the business, compared to the “about us” page. That information can be provided to the dev lead as they assign tasks to developers. But more interesting, that information can be fed into the test scheduling brain, in such way that code commits are followed by more extensive testing.
In summary, the test industry is expected to meet the cadence of dev to allow the business to success with innovation and velocity. However current state is significant investment in scripting, configuration and execution. With modern technology, it is possible to produce the necessary testing from production data and in-sprint requirements, certainly with enough coverage to be meaningful to shift the focus to finding and managing defects.
