Software quality assurance engineers are often under the pressure of cutting down the testing time to ensure on-time product releases. Usually, this means to execute fewer test cycles which may risk the quality of releases, especially in a set up where there are huge impacting features. So making an automated tests faster is the need of an hour, and it is very critical to customise the test execution time with the growing number of features. As the product cycle embraces a continuous integration (CI) that requires a frequent build and test cycles, the pressure to speed up automated testing is intense.However, there are different ways that the QA team can apply to make the test execution efficient and faster.
In TravelTriangle, the QA team has reduced the test execution time by 50% which helps us to cope up with the ever-increasing regression scope.
Let’s discuss the strategies in the following discussion which is generic and applicable for any tech QA team.
Not in the scope of discussion – HOW (code implementation).
1. Parallel Test Execution
Running tests over multiple machines in parallel, which is the game changer in reducing test execution time. Selenium Web-driver GRID is one of the standard options to have parallel test execution, and if team require some customised way to segregate the tests and its execution then this can be attained through customised framework code with basic test automation tool adhering to client-server protocols. A typical structure followed in TravelTriangle is as follows –
Where M1 is the client machine having a code and scripts.
M2,M3,M4,M5 are the server machine where automation tool server(like selenium server and so on) is running.
So, Test Conf 1,2,3… are the configurations to execute tests in M2, test conf 1 can be like “Desktop Sanity tests on Chrome browser on QA1” and test conf 2 can be like “Mobile Regression tests on android device on QA2” and so on. These test configuration will give a lot of freedom in customising the test execution.
2. Test Segregation
Segregating tests execution on different machines in a remote execution setup plays a vital role in subtracting the bottleneck of test build in CI/CD pipeline setup. So the idea here is to segregate the test cases in such a way that no test groups should wait for test execution of other groups aligned on different remote machines. For Example – suppose we have 3 test groups
– ‘Sanity’ take 30 min aligned to machine M1
– ‘G1(Ops + Payment)’ take 70 min aligned on M2
– and ‘TTSN’ take 30 min aligned on M3.
If we run these group in parallel then overall CI pipeline will wait for M2 execution to complete and overall delivery pipeline time would increase.
But, if we change it a little bit like divide G1 into two groups ‘Ops’ and ‘Payment’ further.
- “Sanity + Payment” mapped to M1 which takes 50 mins.
- “Payment 1” mapped to M2 with total time of 45 mins.
- And “TTSN + payment 2” mapped to M3 with total time 45 mins.
So total execution time will be now 50 mins instead of 70 mins.
3. Headless test execution
So headless testing is a way of running browser UI tests without the head, which in this case means that there’s no browser UI, no GUI of any sorts. All known web browsers supports headless test execution integrated with web-drivers. Some popular drivers and library that support is chromium, firefox, HTMLUnitDriver, Phantomjs and so on.
4. Use wait time smartly
A flaky test keeps failing because the back end didn’t respond in time or some resource is still loading, so we put a sleep statement in. Which creates a compounding effect in test execution time. Use conditional waits instead, so that your test does not wait un necessary. Refer this, to understand the waits.
5.Create short but valuable test suites
Choose the most important tests and pull them into a smaller suite that runs faster. These are usually very gross-level tests, but they’re necessary to qualify your system or app for further testing. If these tests don’t pass, it doesn’t make sense to proceed.
6. Let’s also discuss some test framework customisations that supports in reducing the test execution time –
- Halting a test cases at appropriate places instead of running the whole test cases. Optimised test framework should automatically manage the test execution and device proper failure strategy. For example, consider a hypothetical scenario where user fill the form and submit it to register into application, complete the profile and upload all required documents. Now let’s suppose the test case fails while submitting the form, so there is no point of executing steps to complete registration and uploading documents. Now let’s consider a scenario where user lands on listing page and verify different filters, search results and details of items listed. Test case should not halt in case any step fails, as everything should be verified on that page.
- Use APIs and web services to perform user actions wherever applicable instead of navigating it through the web UI. For example, login through APIs wherever test scenario does not really tests login functionality, which will save time.
Hope this discussion helps and QA teams can work more aggressively in reducing the test execution time and reduce the bottlenecks in product deliveries. However, the implementations can differ for different teams based on the test framework designs. Thanks!
Comments