Robustness is the quality of being able to withstand stresses, pressures,
or changes in procedure or circumstance.
A system, organism or design may be said to be “robust”
if it is capable of coping well with variations
(sometimes unpredictable variations) in its operating environment
with minimal damage, alteration or loss of functionality.
Test Script Robustness
Similarly, for Testing Script we may define Robustness as
- the quality to cope with errors during execution (i.e. catch, handle, report, recover)
- the ability to perform despite abnormalities and/or variations in system-under-test
- the ability to recover from total failures in separate testing components
- the quality to be adapted to new test environment in minimal time / with minimal effort
- the quality to withstand a wide range of software/hardware performance issues without losing functionality and/or reliability of testing
- the ability to accept data from various sources with no or minimal changes in source code
Why it’s extremely important?
Let’s see what happens if Testing Script poorly fulfills Robustness requirements defined above.
Execution stops at any point error happened. After the error is fixed, the execution has to be restarted PLUS system under test has to be restored to the initial state.
No reporting – missed bugs!
Unlike people, a program can never “go back”. So, if by the moment when testing script was going to perform a GUI operation on a control, it was not available (did not appear yet, was disabled, was not populated with data, etc.), the script will report the operation as failed. (In worst case scenario, the script will stop, and the entire execution flow will be broken). Read dedicated post.
Poor run-time judgment.
If a script exits on a first legacy fail, how much of coverage is lost? If verification is based on hard-coded checkpoints, how much of manual effort validating numerous mismatches required (don’t forget about maintenance!)? Functionalities, based on the on-screen application response or dynamically changing data could not be verified through a hard-coded checkpoint.
If a script is oriented to verification of a pre-described GUI it has very low observation capabilities which impacts quality and coverage of automated testing.
Primitive Data Models.
Even if no one cared to implement Data Model mechanism, it still exists as a chaotic thing. Importance of having it organized grows exponentially if you plan continuous use of Testing Scripts in changing environments (evolving application, multiple test environments, etc.).
What can you do to improve Robustness?
Upgrade / enhance your existing test automation framework. Functionalities, enhancing Robustness, are re-usable and business / test logic independent.
Move away Service Functionalities (see more in Terminology) from your scripts. Unify the way your scripts interact with GUI, retrieve / store data, report information.
Implement Table- or Tree-based Data Model. Don’t use variables for storing business / test data.
Increase modularity of test logic components and their independence from each other.