Parent page: Test Automation Problems

Robustness

 Automated Testing is about delivering test results – on time, within the Test Phase and reliable enough, with no manual retesting. There is no or minimal time reserved for scripts fixing, debugging and rerunning during the Test Cycle. If Automated Testing didn’t work then QA resources have to perform testing manually. With the above given, implementation better be robust enough to handle unexpected occurrences delivering the most of possible coverage. At the end of the day, test cases are supposed to fail, aren’t they?

 While implementing testing script, automation developer has the following 3 options: ignore error-handling, implement error-handling, implement both exception and error handling. The specific of exception handling in automated testing is in a non-code fails: when application doesn’t behave accordingly to the expected test flow (typical example: unexpected pop-up dialog).

Implementation and problems 

1)     No error handling 

• Whenever error occurred testing tool stops execution. Possibly a defect could be exposed at that point but 99% more likely it’s just a problem with the script 
• After the fix is applied execution could be restarted. However that means time scheduled for testing is used for script maintenance 
• Since at any point test execution could be broken scripts are set up and executed manually one by one. Test results are also validated manually. With all the manual involvement it becomes no longer really automated 
• A continuous requirement going into the code means requirement keeping a code-familiar person along with the scripts. And knowledge transfer and learning curve problems in case of transitions 
• At the end, having low robustness and high maintenance cost scripts means keeping twice as expensive automated testers doing things neither better nor faster than manual testers 

2)     GUI verification and error-handling 

• Verification of GUI (e.g. whether the button exists and enabled before you want to click it) before performing operations allows more accurate coverage and more robust test execution. However, in case of unexpected changes in the flow, like a pop-up dialogs, test case is failed without a true reason 
• Verification, error-handling, and reporting make every single test step implementation more complicated from the coding perspective. Thus more advanced programming skills are required from test automation person 

3)     Verification, error-handling and exception handling 

• Verification of GUI, handling and recovering from all errors, test flow exception handling allows unbreakable and unattended test execution – and that provides delivery of test results on time with the great coverage. However, such a quality level is achievable only with a very seasoned automation professional