In our practice, software test automation can be an extremely powerful tool which if implemented properly can help ensure a successful testing engagement. But the fact remains (as a lot of you may know first hand) that when not managed properly the use of automated testing tools will be a complete waste of time and energy and can even create substaintally more problems than solutions for the project.
It is important to point out that it is not always necessary to automate all
functionality of the application under test. In fact, based on my experience with software quality assurance it rarely makes sense to do so. Usually it makes the most sense to automate the “happy path” test cases, core business rules validation and critical application functionality, and any repetitive tasks or test cases that must be duplicated thousands of time. If the remaining functionality is relatively easy to manually test, and it will take more effort to develop automated test scripts, then continuing with manual testing.
The following list outlines some guidelines that can be used to determine when it is a good idea to introduce automated testing into the QA process of a software development project.
Test Cases Execution and Validation
At the start of each project we analyze the system from the automation tool perspective to determine the difficulty level of test cases automation. Projects are evaluated on GUI elements and overall GUI complexity, as well as identifying the applications use of validation patterns. The more consistent the validation patterns, the easier and more effective test automation will be.
Repetitive Tasks
Repetitive tasks which can have multiple outputs are also very good candidates for automated testing. An example of this would be testing application security, permissions, and user roles.
Ensure Consistency
Test cases that must always be run following very precise steps are excellent candidates for automation, as automation will ensure that a test is always run following a consistent set of exact steps during each execution.
Testing Data Generation
Test automation can be used for generating the test data required for both manual and automated testing.
Testing Staff Skills
An important matter to consider is the overall skill set and technical abilities of the testing team. Automation is obviously a better choice when you have access to a highly technical testing staff. Conversely if you have access to a highly successful (yet not very technical) manual testing team it is better to continue to take advantage of the manual execution as opposed to putting in the extra effort required while attempting to automate the test suite.
Project Planning
Depending on the project schedule and estimated project lifespan (are multiple versions anticipated or is this a 'one-off' project) you can determine if test automation is a good solution. Developing an automated test suite involves a substantial investment in time and effort -- so a one-off project with a short develpment schedule is not the best candidate for automated testing. Whereas a longer project that is anticipated to have a longer lifespan would be a much better fit.
Number of Module and System Integrations / Regression Testing Needs
When a development project has a large amount of module and system integrations regression testing is extremely important to ensure that all exiting functionality remains in working order. If developed properly, having an automated test suite is an excellent solution to ensure fast and consistent regression testing with each test cycle.