Attendance

Automated tests must be designed for unattended execution.

Failure criteria:

• Test script may stop or break execution at any moment. Manual action required to continue the execution. Full restart is required if execution was stopped.
• Upon error or failure execution is skipped to the next script, leaving gaps in coverage.
• Scripts don’t log errors or failures.

Robustness

Robustness is a capability of automation to withstand execution faults and adapt to fluctuations of test environment with a minimal impact on test coverage during the execution.

The following requirements must be implemented for the all scripts.

• Error-handling – all faulty operations (i.e. all GUI operations, file operations) must have proper code for positive and negative outcome. All potential errors and faults must be detected and properly reported by the script.
• Upon insignificant failures, execution of the test case must continue to provide the full coverage possible.
• Upon errors and “stopper” failures, execution of the test case must be finalized, problems must be reported by the script, and execution should continue with the next test case. If subsequent test cases are destined to fail upon failure in the pre-requisite test case, then execution must skip them and report accordingly.
• Synchronization – all scripts must execute the workflows in synchronization with the application’s behaviour. This includes proper waiting for GUI reloading or failure upon time-out if it didn’t happen.
• All timeouts must be specified as global parameters, manageable from the central source, and defined as properties of the test environment.

Failure criteria:

• Test suite cannot be pointed from one test environment to another without code changes.
• Execution errors are not reported.
• Over 1% of “false alarms” – failures occurred due to synchronization or other script problems, whereas the application was stable.