Original date: 21 Apr 2009, 12:30pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Usability Requirements Matrix 2

Transparency
– Do you want Automated Test execution to be another “black box”, or you want to have a fully reproduced picture with documented test results?

Execution Log

Built-in execution log provided by the Testing Tool and not accessible outside. Contains no Business Logic details. Test Case irrelevant.
Some tools, like QTP, log pretty much everything. Some other log only a high-level component calls. And most of them do not provide a convenient execution report embedded from the Testing Tool itself. A few that do still don’t have it Test Case oriented.

Execution Report

Embedded from Test Tool. Contains information of GUI steps performed along with the data used. (Possibly) contains verification details. Does not have validation details. May not be Test Case relevant.
Some good Testing Tools allow exporting of Execution Log into Web-page, PDF-, or Word-document format. At least, it allows sharing test results with other team members who don’t have Testing Tool installed.
However, due to lack of logging Logical steps (validations, judgments) Test Flow is still not detailed enough to get a full picture without reproducing Test Case manually. Another reason could be a lack of implementation. Since logical judgments and validations are not implemented they’re not reported. And automation developer still must take care of making Execution Report Test Case oriented.

Test Report

Embedded from Test Tool. Contains detailed information on GUI and Logical (Verification/Validation) steps performed providing end-user with the fully reproducible Test Flow. Optionally may contain Fail reports and other reconciliation details.
This is something looking like as it’s done manually but better (because manual tester will get really bored accurately documenting hundreds of Test Cases executed).

 Operation
– Do you want to have special stuff executing Automated Test Cases or you need a “click-and-start” procedure?

Developer Only

Only automation developers can setup and start scripts.
When there is no convenient “front-end”, and all the configurations are part of the code, only the person very familiar with the script can successfully launch it.

User Friendly

Only basic computer skills required to start Automated Test Plan. Short training required for the unified setup procedure.
HP/Mercury Quality Center is one of the perfect examples.
However, QC doesn’t provide much customization features automatically applicable to the batch you’re starting. So if it’s something more complicated than a set of independent Test Cases executing on a single environment, advanced design and implementation of an Automated Test Plan document is required. But custom development must still strictly follow easy-and-simple-to-use requirements.

Design
– When it comes to using and re-using automated test assets, is it possible for automation developers only, or it’s built convenient enough to be successfully used by non-technical testers without big exposure to automation?

 Developer Only

Only developers can create Test Plans or do modification of existing ones; due to hard-codings everything requires code change.

Specialist Only

Specially trained QA with technical knowledge can do minor modification. Due to all the test logic stored as code development the risk is still high.
Half-way to reach a convenient customization is passed but there is a lot of things still done manually.

User Friendly

Non-technical user (QA, BA) can create new and modify existing test flow using Keyword-Driven front-end.
It could be as simple as editing Excel spreadsheet, or drag-n-drop operations on a visual call tree.
HP Business Process Testing Framework is a nice example of that.

Swiftness
– How quickly the existing Test Suite allows adding (development) new test assets, like Test Cases and reusable testing functionalities?

Individual Creation

Each manual Test Case requires development (or recording) of Test Script. No framework.

Modular Programming

Test Cases developed individually; common functions are maintained in library files. Sub-framework level.

Visual Prototyping

Test Case automation DOES NOT require programming. Framework fully supports Business/Test Logic execution from the back-end.
Hybrid Keyword/Data Driven Framework implementation is required as an initial investment.
Or you can try some of the commercial ones:
HP Business Process Testing (VBScript is a programming language)
Worksoft Certify (doesn’t have programming language support).