Original date: 12 May 2009, 5:00pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Maintainability Requirements Matrix

Setup
– What level of convenience you want to have setting-up the scripts?

If Test Data file and path are hard-coded in the script then pointing to another data file requires code change. If the login procedure has username and password hard-coded then every change of login credentials requires code change, and concurrent use of different login credentials is not possible.

The alternate way is to use configuration file in Excel, XML, or plane text format. If Testing Solution includes multiple scripts (>50) implementing standard template (file structure, format, locations, etc) will quickly payback on maintenance efforts and learning curve for non-technical users.

GUI Change
– What level of efforts and what level of impact are acceptable when considering GUI Mapping approach?

Testing Tools recognize GUI objects (Button, Input Box, etc.) referring to visible (E.g. Button Text) and invisible (E.g. Internal ID) technical (physical) properties. So addressing a specific GUI object requires providing Testing Tool with a unique set of properties. Direct specification of the properties is usually named as “Description” of the GUI Object, and programming with the Object’s properties directly specified is named Descriptive Programming (Refer to QTP Manual for more details). In the same time enlisting of physical properties (E.g.”window id:=1203”) has no or little business value and requires specifying all of them. To increase readability of the code and reduce code maintenance cost most of Testing Tools now offer storing physical descriptions separately and referring them through logical (business) name. While still being the same for the Testing Tool, “Input text to “Username” makes more sense then “Input text to “window id:=1203” when read by a person. Same GUI control used through different scripts by the same logical name still refers to a single entry in descriptions file (named “Object Repository” file in QTP) so the GUI change requires single point adjustment of the file storing descriptions and not “find-and-change” operations in all the scripts.

Business Logic Change
– What level of efforts and what level of impact are acceptable when Business Functionality is changed and scripts are to be fixed/updated in order to bring them working back?

Individual Code changes

Since business/test logic is contained within the code, code change is always required for the all scripts impacted. The higher code duplication is in the scripts the higher is the impact.

Modular code change

Since Test Cases are built from reusable modules covering business/test logic, only code change within impacted module is required. Modular approach allows avoiding duplications of the code and reduces code maintenance effort.

Visual Prototyping

Using Drag-n-Drop and NO PROGRAMMING non-technical users can create/modify Test Components contained in XML file.
Most of Hybrid Keyword/Data Driven Frameworks have a user-friendly Front-End. As the most simple implementation it’s an Excel table, the more advanced utilize XML-based visual tree.
Business/Test Logic change within a window requires update of testing instructions with the effort similar to data maintenance cost and possible impact much less than a direct code change.

Code Maintenance
– Whether it is update corresponding to the Business/Test Logic change or debugging/fixing of the code what level of efforts and what volume of code are acceptable?

Test Code is significantly bigger than Application Code

This volume of code is typical for Record/Playback or Individual Test Case automation.
Extremely high maintenance cost and possible impact on the testing schedule.
If 10 manual Test Cases for the same screen were covered by 10 individual Test Cases, the code duplication rate was probably over 90%.

Code is bigger than application code

This volume of code is typical for Modular / Parameterized Test Case automation.
The bigger number of screens the higher code maintenance cost.
While the code is somewhat structured and test scripts reuse coverage of the same functionalities, the general steps like GUI interactions and verifications are repeated in the code over and over again.

Business Logic Framework

Only the Framework is coded. Business/Test Logic is not part of the code and can be of any size. File versioning tools could be used easily. Duplication does not create an impact.
All the GUI interactions and verifications are coded only once. Covering a wide range of possible operations means bigger amount of service code for a single Test Case. With 20-30 Test Cases the advantage becomes highly visible. With a high volume (100+) of Automated Test Cases the difference is dramatic.

Test Input and Verification Data Maintenance

Verification Data. Hard-coded GUI checkpoints

For the GUI Regression type verifications checkpoints are to be captured for every single GUI screen and every single data row. Extremely high creation and maintenance cost.
Additionally checkpoint fail still requires manual investigation to find out what functionality has a problem.

Verification Data. Individual GUI checkpoints

Checkpoints are captured individually only for the GUI controls required. Change in other GUI doesn’t impact the checkpoint. Verification type is for equality only but the expected result could be parameterized what adds flexibility. High creation and maintenance cost.
GUI control specific checkpoint is better correlated with the functionality testing.

Business Verification Rules

Applied individually to the GUI control OR on-screen functionality. Wide range of verification rules gives great flexibility. Possibility of reusing input data as expected result lowers maintenance cost comparing to the individual GUI checkpoint.
This type of verifications is similar to the manual functional testing and with the proper validation implemented produces test results that do not require manual investigation to find out what functionality has a problem.

Test Input. Data is a part of the script package

Each script has data table coming along with and referred by hard-coded name and path. Code change required in order to change data source. Very inconvenient for testers (can’t change directly, access to the testing tool is required, training is required).
High maintenance cost due to a lot of manual efforts.

Test Input/Output. Independent Data Model

Test Plans/Test Cases refer to data files through mapped logical names. Features full flexibility (keep multiple copies, store on different drives, copy/exchange files) and convenient maintenance. No code changes required in order to change data source.