The Good, The Bad and Test Automation

Software testing is not a new field but is changing at an extremely fast pace that requires QA engineers to update constantly with new technologies, methods and best practices. Online resources, books, papers unfortunately don’t hold impressive information, even prestigious certifications aren’t that useful as some people suggest them to be. After working in this industry for five years now I came across many concepts and interesting theories that are, simply put, not true or implementations that do more harm than good, starting with but not limited to: trainings, agile methods , testing methods, measurements , zero defect demonstration, test automation, exhaustive testing and many more.
Picking from the list a topic at a time, I want to share my point of view and challenge coworkers to a discussion.

Today’s topic on the whiteboard: Test Automation.

Test Automation is often seen as a way to reduce the cost of testing, shorten testing cycles, increase coverage and effectiveness, and in general that this is the silver bullet of any QA department.

Nicely put, but usually wrong.

The trend today is that test automation is enforced on some projects that don’t have good results in an attempt to achieve that nice statement presented above.

Management sometimes can become extremely focused on automation rather than improving quality, I’ve seen projects with estimation that have 90% automation, others that promised the creation of hundreds of scripts per day and the classic case when the QA department doesn’t have time to test so the only viable solution considered is automation.

Don’t get me wrong, I strongly believe in test automation but only where it adds value, and that is the tricky part. Automation testing involves higher upfront costs and should be considered a long term investment where the most important variable of the equations is the return of investment. Usually the break even point is considered to be somewhere around 1-3 years.
In the end it’s all about the money and the QA department has a big responsibility here.

Here you can find an interesting tool that helps calculate ROI: http://www.aspiresys.com/testautomationroi/

As you can see in the ROI calculator above, the decision to make a jump from manual to automated testing is a big step and should take into consideration a lot of parameters, such as:

• Age of the product
• Releases planned for a year
• Required regression cycle per year
• Number of existing regression test cases
• Number of configurations to be tested
• Cost of the testing resources
• Infrastructure cost
• Test automation tool cost
• Percentage of test cases that can be considered for automation
• Time required for building reusable components, automation frameworks and batch scripts
• Hourly cost per test automation resource
• Resource training

 

 

 

 

 

 

Test Automation is not something that can be done overnight so before starting, try to have an in depth overview of the project you are working on.

Another thing that usually is not taken into consideration when deciding if test automation is feasible is the product development life cycle duration. If you are working on an application that will be developed and tested in 5 months, automation could be a mistake.
Keep in mind that when effort is invested to write automated test, you implicitly lose time that could be used to find and report defects. Contrary to popular belief automation needs constant updates and debugging especially in an environment where developers are not used to working together with the QA team, where everyone writes code without following a formal procedure.

In my experience more defects are found via ad-hoc (exploratory) tests than via automation. The more time a tester spends playing with a feature the greater the odds of finding real user bugs. An automated script will check a specific flow, specific point in the application depending on what you specified when you wrote it. Don’t underestimate how stupid automated tests can be. A manual check will take into consideration a lot more aspects of the application, things like patterns, layout, complex functionalities, defect clustering and other features.

One other thing I would like to draw attention to is the “Pesticide paradox”. A lot of engineers don’t give this test principle the appropriate attention, they will just be happy that their automated test suite shows no issues.

Pesticide paradox test principle states that running the same tests over and over again would not show any new defects because all defects found using the initial test cases would eventually all be fixed.

Therefore, to avoid the pesticide paradox and other causes of diminishing returns, all tests should be regularly reviewed to ensure that all expected areas of functionality are covered.  This could involve adding new scenarios as well as discontinuing ones that are no longer valid.  Additionally, new tests can be written to exercise the code in new or different ways to highlight potential defects.  Finally, use a mix of formal and informal testing techniques to achieve more holistic testing results.
source

Another mistakenly qualified “best practice” is that test automation should be done in the beginning of the test cycle. Sounds good on paper and the principle is correct, but it rarely works in real life. At the beginning of the project things are changing fast and they usually tend to be chaotic, my advice for the best time to automate tests is somewhere around the middle of the cycle or to better put it: when things are mostly stable but a lot of builds are left to test.

Here are some areas where automation has great value:

• File interaction (open/save/export)
• Authentication
• Security holes
• Core/critical area functionalities (functionalities that the application revolves around)
• Features/functionalities that don’t use a dynamic oracle.
• Important scenarios that have to be performed often
• Data comparison

To conclude, automated testing is not and should not be considered a silver bullet that will magically solve all the problems of testing and software development. It is a valuable tool in your toolbox and if used wisely it can enhance the overall quality of a product, but the key ingredients of any successful project are: the right level of automation combined with exploratory testing and good test design.