Not a Load Test Plan template

(An example of a situational feedback approach)

 The Law of Large Numbers theorem states that the average of results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed. Unfortunately, application’s behavior under a load abides much more casual Murphy’s Law. To make it completely bad, in testing we have to consider worst happened case. That is, if 99 users (of 100) were successfully able to pay their bills online and only one got transaction lost once, the payment application is nowhere near 99% usable, and we have to catch somehow this error that didn’t occur during functional testing. 

Load Testing - simulation of concurrent user activities and investigation of application’s behavior under such circumstances.

So, what would you want to test?

Do you want to verify everything?

- This might take twice as long time as your time before the release. Or maybe 10 times longer.

So, how could you plan load testing?

In my testing experience I had both projects when I was asked for specific answers (E.g. page response time) and where I was solely responsible for testing strategy and approach; when the all activities were planned and scheduled and where testing was performed ad hock and upon request. Based on those experiences, I suggest to assess questions ”What can you do?” and “What results will be of [immediate] value?” before designing a Load Test Plan. Answers to first question heavily depend on availability of Test Environment and testing tools (and, of course, skills of a tester). Answers to second question are context-specific: they depend on the project type and project phase.

If your Test Infrastructure is not as powerful as Production Infrastructure? 

Importance of having Test Environment equally powerful (and with identical infrastructure) as Production Environment is critical. However, sometimes Test Environment is not always available, is not yet available, or just not available due to a variety of reasons.

What is worthless to do?

Any thresholds and bottlenecks found will not match with those on Production Infrastructure. Any back-end performance stats will not match with those on Production Infrastructure. If performance of some infrastructure components of the same type (for instance, database) is comparable then high volume tests or stress tests could give some relevant information but still very limited.

What can you do?

Front-End application performance is performance of a client-side part of it. If a web page is rendered slowly in a browser then it won’t matter how fast infrastructure is. So you can investigate front-end performance even with poor infrastructure and without generating a load.

Most of web-based applications are supposed to be accessible 24/7. On a back-end that requires endurance and robustness from application servers and databases. Endurance for a server means careful management of available resources and robustness means ability to recover after errors. If an application server side has problems with RAM management then memory leakage will eventually slowdown or crash it. More powerful hardware can only prolong up-time.

If you are a part of application development team? 

Earlier detection of errors gives more time for fixing them. What kind of feedback on application’s performance is the most appreciated during development phase?

Front-End Performance

Same implementation approach is usually shared across the application. Any page performance issues, discovered early in development, might have an application design level positive impact.

Back-End Performance

Load/Performance testing of back-end components and entire infrastructure is often used with the main reference to a database performance testing. While many “bottlenecks” are truly often found within database, the following investigations should be conducted as well.

• An application server performance and behavior during different types of load (stress, spikes, etc.)
• How load balancing module affects performance?
• How anti DDOS module (or any Firewall) affects performance?
• How encryption affects performance?
• What level of tolerance has an application to latency, packet loss, link faults and other dynamic routing effects common for dial-up, ADSL, WAN types of connection?

Endurance and Robustness

Most of modern programming platforms have memory management functionalities hidden deep inside and not directly controlled by developers. Monitoring performance of application servers helps gathering information on efficiency of so called “Garbage Collectors”, “Thread Classes”, and other components of an application that do not carry business functionality but may have a direct impact on it. Any other resource allocation and management problems if discovered earlier will help saving time and increasing the potential performance of an application.

Scalability Tests

Once components of an application are linked and functional (usually, that is System Integration Testing phase) scalability tests will help to identify performance thresholds of an application. Test results will be meaningful if the Infrastructure -Under-Test (or, at least, its tested components) is similar to Production Environment, and load levels are equivalent to or exceed expected load in Production.

If the project goal is deployment of a vendor provided application? 

You may still complain about defects found in off-the-shelf software, and they will eventually get fixed (most likely, in the next release) but implementation project can’t wait that long. It is critical to find out as much as possible about all instability thresholds, possible vulnerabilities – and test how well they could be covered by external measures, such as load balancing or filtering tools, expansion of hardware or network resources.

So, what tests should you target first?

Scalability and Performance Tests

If the infrastructure is already designed and implemented it’s really critical to find out whether its capable of carrying the expected load and what capacities it has. If any performance requirements are defined (or provided) by stakeholders you may conduct performance compliance tests. Otherwise you may capture performance characteristics at the highest level of load where application is still stable and provide stakeholders with that information. Those performance characteristics might also be used in regression/comparison testing whenever application or infrastructure change in the future.

Stress and Endurance Tests

Finding out how infrastructure behaves under extensive load or during peak periods is a next priority task. Issues found might not be fixed programmatically but still could be workarounded and definitely need to be identified. It also could be not a one time job if you’re helping in configuring of load balancing tools or firewall software.

Conclusion

I am a firm believer that the role of documentation is critical in testing. However, I value beforehand created documentation much less than structured records of how it actually was happening. I also often prefer digging extra time around issue(s) found rather then proceeding along with a plan. I always plan but my plans are essentially goals and milestones to reach rather than steps to walk. I prioritize them on the flow, based on real-time information received and events that have already occurred. This is a situational feedback approach.

Do you prefer Procedure Plans or Goal Planning?

Original date: 12 May 2009, 5:00pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Maintainability Requirements Matrix

Setup
- What level of convenience you want to have setting-up the scripts?

If Test Data file and path are hard-coded in the script then pointing to another data file requires code change. If the login procedure has username and password hard-coded then every change of login credentials requires code change, and concurrent use of different login credentials is not possible.

The alternate way is to use configuration file in Excel, XML, or plane text format. If Testing Solution includes multiple scripts (>50) implementing standard template (file structure, format, locations, etc) will quickly payback on maintenance efforts and learning curve for non-technical users.

GUI Change
- What level of efforts and what level of impact are acceptable when considering GUI Mapping approach?

Testing Tools recognize GUI objects (Button, Input Box, etc.) referring to visible (E.g. Button Text) and invisible (E.g. Internal ID) technical (physical) properties. So addressing a specific GUI object requires providing Testing Tool with a unique set of properties. Direct specification of the properties is usually named as “Description” of the GUI Object, and programming with the Object’s properties directly specified is named Descriptive Programming (Refer to QTP Manual for more details). In the same time enlisting of physical properties (E.g.”window id:=1203”) has no or little business value and requires specifying all of them. To increase readability of the code and reduce code maintenance cost most of Testing Tools now offer storing physical descriptions separately and referring them through logical (business) name. While still being the same for the Testing Tool, “Input text to “Username” makes more sense then “Input text to “window id:=1203” when read by a person. Same GUI control used through different scripts by the same logical name still refers to a single entry in descriptions file (named “Object Repository” file in QTP) so the GUI change requires single point adjustment of the file storing descriptions and not “find-and-change” operations in all the scripts.

Business Logic Change
- What level of efforts and what level of impact are acceptable when Business Functionality is changed and scripts are to be fixed/updated in order to bring them working back?

Individual Code changes

Since business/test logic is contained within the code, code change is always required for the all scripts impacted. The higher code duplication is in the scripts the higher is the impact.

Modular code change

Since Test Cases are built from reusable modules covering business/test logic, only code change within impacted module is required. Modular approach allows avoiding duplications of the code and reduces code maintenance effort.

Visual Prototyping

Using Drag-n-Drop and NO PROGRAMMING non-technical users can create/modify Test Components contained in XML file.
Most of Hybrid Keyword/Data Driven Frameworks have a user-friendly Front-End. As the most simple implementation it’s an Excel table, the more advanced utilize XML-based visual tree.
Business/Test Logic change within a window requires update of testing instructions with the effort similar to data maintenance cost and possible impact much less than a direct code change.

Code Maintenance
- Whether it is update corresponding to the Business/Test Logic change or debugging/fixing of the code what level of efforts and what volume of code are acceptable?

Test Code is significantly bigger than Application Code

This volume of code is typical for Record/Playback or Individual Test Case automation.
Extremely high maintenance cost and possible impact on the testing schedule.
If 10 manual Test Cases for the same screen were covered by 10 individual Test Cases, the code duplication rate was probably over 90%.

Code is bigger than application code

This volume of code is typical for Modular / Parameterized Test Case automation.
The bigger number of screens the higher code maintenance cost.
While the code is somewhat structured and test scripts reuse coverage of the same functionalities, the general steps like GUI interactions and verifications are repeated in the code over and over again.

Business Logic Framework

Only the Framework is coded. Business/Test Logic is not part of the code and can be of any size. File versioning tools could be used easily. Duplication does not create an impact.
All the GUI interactions and verifications are coded only once. Covering a wide range of possible operations means bigger amount of service code for a single Test Case. With 20-30 Test Cases the advantage becomes highly visible. With a high volume (100+) of Automated Test Cases the difference is dramatic.

Test Input and Verification Data Maintenance

Verification Data. Hard-coded GUI checkpoints

For the GUI Regression type verifications checkpoints are to be captured for every single GUI screen and every single data row. Extremely high creation and maintenance cost.
Additionally checkpoint fail still requires manual investigation to find out what functionality has a problem.

Verification Data. Individual GUI checkpoints

Checkpoints are captured individually only for the GUI controls required. Change in other GUI doesn’t impact the checkpoint. Verification type is for equality only but the expected result could be parameterized what adds flexibility. High creation and maintenance cost.
GUI control specific checkpoint is better correlated with the functionality testing.

Business Verification Rules

Applied individually to the GUI control OR on-screen functionality. Wide range of verification rules gives great flexibility. Possibility of reusing input data as expected result lowers maintenance cost comparing to the individual GUI checkpoint.
This type of verifications is similar to the manual functional testing and with the proper validation implemented produces test results that do not require manual investigation to find out what functionality has a problem.

Test Input. Data is a part of the script package

Each script has data table coming along with and referred by hard-coded name and path. Code change required in order to change data source. Very inconvenient for testers (can’t change directly, access to the testing tool is required, training is required).
High maintenance cost due to a lot of manual efforts.

Test Input/Output. Independent Data Model

Test Plans/Test Cases refer to data files through mapped logical names. Features full flexibility (keep multiple copies, store on different drives, copy/exchange files) and convenient maintenance. No code changes required in order to change data source.

Original date: 6 May 2009, 12:30pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Scalability Requirements Matrix

Platform
- What applications should be able to handle the Framework?

The business need for automation of QA/testing activities appears when testing volume, turnover and coverage requirements make it beneficial enough. Volume means a big number of test cases which means a big set of functionalities. (High) turnover and (wide) coverage requirements mean a demand for test results within a tight timeframe and with the specified (high) level of extent.
With the mentioned above it comes to Enterprise level software products, tested from the front-end and back-end, and often having different front-ends: Windows Desktop, Standard Web-based, Rich Web App (Web 2.0 and higher), and UNIX-based accessible from Terminal Emulator (Text-based Screen Interface).

Upgrade
- When requirements are extended or revisited what efforts required and which way the upgrade of testing scripts supposed to be performed?

Rebuild
Upgrade requires rebuild from scratch. Testing is not available until it’s done.
Individually developed (or recorded) test cases could be upgraded only through rebuild.

Replacement
Upgrade requires replacement of a module. Testing is not available during replacement only.
Structured (modular) development allows keeping common functionalities in reusable function libraries. Change in the login procedure, for example, requires change of a common function implementing the procedure without impacting all the scripts where it’s used. For the new functionalities, like additional verification requirements specific for the screen, new reusable function is needed to be implemented and called from the scripts covering the screen.

Plug-in / Plug-out
Upgrade/Replacement/Adding new Test Component DOES NOT bring testing down. No code change is performed at all.
Further evolution of Automation Frameworks allows automating the automation itself. Hybrid (Keyword/Data Driven) and Business Model based frameworks bring Test Case automation to the level of abstraction where no scripting is required (record/playback is not required either!). Operating by high-level instructions (or keywords) Business SME can design/modify a Test Case (See Visual Prototyping/Swiftness/Usability requirements matrix).

Migration
- When time comes to migrate to a new platform (new Testing Tool) do you want to keep your assets or you have to rebuild everything from scratch?

Rebuild

If all the logic was contained in the code, migration to a new Testing Tool will require rebuild from scratch.
Having proper documentation support of the automation implemented (Test Flows, Coverage documents, Maintenance guides) allows reducing risks and expenses of the rebuild-from-scratch project.

Framework

External Business/Test Logic could be supported by any Testing Tool. Only Framework implementation is required.
Frameworks having support for externally customizable Business/Test Logic (Keyword-driven and higher on evolution curve) allow keeping Business Test Assets gained. It’s not only saving time for the automation development expenses but also saves a lot of analysis and design time, that has been invested creating automation.

Volume
- When the volume of dataset could change significantly will it require rebuilding the scripts or only some external customization is required?

Hard-coded volume

Test scripts are built to use only certain volume of data.
Most typical examples here would be a script verifying only 1 row in the report or a script submitting only 1 record through a multiple request form.

Unlimited volume

External customization allows using of scripts with any volume of data required. User can specify desired number of data records as a comma-separated list or as a certain rule (e.g. “Find and verify all report rows matching the account number”).

Functionality
- Does the Framework support functionalities below, matching, or above current needs?

Very often automation project is started as a pilot. After the certain phase, “Proof of Concept” gives management an exposure to decide whether they see it worth to continue or automation doesn’t worth funds invested.
Or the initial scope of automation (“just 10 Test Cases“) does not presume capabilities to extend for the framework.
As a result quickly baked framework is not capable to support extending business and automation requirements. This causes either downsizing the requirements or unstable, unreliable work of test scripts. In both cases automation is considered a “barely useful”.

Original date: 28 Apr 2009, 1:00pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Robustness Requirements Matrix

Manned / Unmanned Test Execution
- What level of manual efforts in automated testing would be acceptable for you?

Baby-sitting

Test script may stop or break execution at any moment. Manual action required to continue the execution. Often full restart is required.
This is typical for all the scripts created using any variation of Record/Playback method. Furthermore, using of standard GUI functions without prior GUI verification and synchronization gives similar problems.

Skip errors

Test script doesn’t break the execution on error. However skipping to the next step (or Test Case) leaves unreported gaps in coverage. Manual investigation of the cause is required.
Test Plan wise execution is a quite robust because one failed Test Case doesn’t stop the whole sequence. However, at Test Case level such an implementation seriously reduces coverage. For long flow test cases visiting 5 and more screens exit on the first error occurred at the first screen means leaving over 50% of Test Case uncovered.
While testing manually, minor fail requires documentation but doesn’t require stopping of testing activity on the screen.

Handle errors

Test script doesn’t break the execution and handles the errors along with automatic investigation and reporting.
Implementation should include catching internal errors and Test Flow exceptions (pop-up dialogs and other unexpected events interrupting the normal flow), recognition of the problem, investigation and recovery steps along with the detailed reporting.
Manual testing example.
1) Select payment currency USD (dropdown listbox).
2) Select payment type WIRE (dropdown listbox).
3) Press “Make Payment” (button).
4) Verify payment details on the Confirmation screen.
Possible fails at step (1)
1A. Failed to find dropdown. Severity – High. Can continue testing? Yes. Report absence of the object.
1B. Failed to use dropdown because it’s disabled. Severity – Medium. Can continue testing? Yes. Report state of the object.
1C. Failed to select item USD in the list. Severity – Medium. Check other items available (CAD, AUD). Report absence of required item and other items available. Can continue testing? Yes.

Possible fails at step (2)
2A. Failed to find dropdown. Severity – High. Can continue testing? Yes. Report absence of the object.
2B. Failed to use dropdown because it’s disabled. Severity – Medium. Can continue testing? Yes. Report state of the object.
2C. Failed to select item WIRE in the list. Severity – Medium. Check other items available (CASH, CHEQUE). Report absence of required item and other items available. Can continue testing? Yes.

Possible fails at step (3)
3A. Failed to find button. Severity – ShowStopper. Can continue testing? No. Report absence of the object.
3B. Failed to press button. Severity – ShowStopper. Can continue testing? No. Report state of the object.

At step 4 (as well as before going into step 1), absence of screen (Window, Dialog, or Web-page) is a ShowStopper fail for the script.

Now what if script exits at the first fail encountered? Even with 4 simple steps, we have coverage reduced to 20% of a possible one when same testing is done manually!

Automated Decision-making
- Does the script only GUI input, or verification, logical judgment, test status evaluation, and overall validation of the Test Case?

GUI steps only

With no verification implemented, GUI data entry has low relevance to testing.

GUI steps + Window checkpoint

Verification is based on hard-coded Window checkpoint. No logic (only equality). No validation (how bad is the mismatch?).
It used to be very popular and seemed to be good workaround reaching detailed level of verification without implementing verification logic. However, huge data maintenance effort popped-up very quickly as every single data change causes checkpoint to fail. And the validation of test results still had to be performed manually which for complicated checkpoints is comparable to the manual testing time of the application itself. Another problem has been revealed through a static nature of this verification type. Functionalities, based on the on-screen application response or dynamically changing data could not be verified as a hard-coded checkpoint.

GUI steps + verification

Each GUI control is verified with an individual parameterized checkpoint. No logic (only equality). No validation (how bad is the mismatch?).
Splitting one consolidated window checkpoint into individual GUI control checkpoints allowed reducing maintenance efforts. Some tools (like QTP) allow parameterization of expected result which improves maintainability even more. However, verification is still based on “equal or not” rule only, while business requirements may say “equal with rounding”. “greater/less than”, or even “includes / included”. And validation of comparison is still performed manually.

Business Verification Rules

Externally customizable verification/validation rules applicable to every GUI or Logical step in the Test Flow.
To reach this level of convenience, Hybrid or Model based Framework should be in place. In addition, the Framework should have a user-friendly “front-end” allowing design and customization of verifications in the Test Flow without programming.

Synchronization
- Do you want an automated Test Flow to be successfully running on different computers, different networks and environments?

No sync.

Test script does not wait for the application / network to catch up. Missed / failed steps as a result. Modern testing tools are trying to minimize fails by setting an overall timeout applied to the all GUI steps by default. When synchronization time fits the default timeout (usually 30-60 seconds) the problem is resolved. However, the major steps (login, submitting report, creating new record) may require significantly longer waiting time affected by the network speed and server performance.

Hard-coded sync.

Wait time is defined as a hard-coded value during script development phase. Unstable work on different computers / networks as a result.
Example.
During script development phase, it was taking about 3 seconds for the report window to appear. “Wait (5)” instruction was put giving 5 seconds for the window to appear. However, during test execution phase certain reports required much longer time to be generated and all that test cases were failed by the script.

Time-out based

Wait time is customizable individually per GUI step; global time-out parameter allows adjusting all sync’s for the network speed.
Most of Testing Tools allow using various synchronization statements. However, that code is not generated automatically during recording. It must be set programmatically before GUI instruction or as a part of wrapping custom GUI function.
The difference between “Wait(30)” and “Sync(30)” is in an exit condition. In the first case, the script will just wait doing nothing for 30 seconds.
In the second case, the script will be checking the GUI to become available for the period of up to 30 seconds.

Data Model
- How flexible Test Automation Framework is to support test data of a various, customizable volume, and to be quickly adjustable for the new test requirements?
The general difference is in the automation approach. In “manual automation” every manual test case is automated individually, which gives convenience for “automation developer” but creates numerous duplications in code and data, which produces a great inconvenience using and maintaining such scripts. In Test Flow Automation multiple test cases using same screens and having same major PASS/FAIL conditions are covered with single script with test data controlling the flow (data-driven).

Variables / Single source

Test script uses values from variables initialized from the single data sources. Any change in logic, data structure, etc. produces heavy impact and requires redesign of the code.

Datatable / Single source

Test script uses values referring by logical name in the data table imported from a defined source (external data spreadsheet). Changes in logic and data structure could be reflected with low/medium effort. Changes in data source still require redesign/modification of the code.

Independent Data Model

Test script uses values referring by logical name in the internal data space. Changes in logic and data structure could be reflected with low effort. Changes in data source do not require script redesign – only substitution of another initialization module is needed.

Original date: 21 Apr 2009, 12:30pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Usability Requirements Matrix 2

Transparency
- Do you want Automated Test execution to be another “black box”, or you want to have a fully reproduced picture with documented test results?

Execution Log

Built-in execution log provided by the Testing Tool and not accessible outside. Contains no Business Logic details. Test Case irrelevant.
Some tools, like QTP, log pretty much everything. Some other log only a high-level component calls. And most of them do not provide a convenient execution report embedded from the Testing Tool itself. A few that do still don’t have it Test Case oriented.

Execution Report

Embedded from Test Tool. Contains information of GUI steps performed along with the data used. (Possibly) contains verification details. Does not have validation details. May not be Test Case relevant.
Some good Testing Tools allow exporting of Execution Log into Web-page, PDF-, or Word-document format. At least, it allows sharing test results with other team members who don’t have Testing Tool installed.
However, due to lack of logging Logical steps (validations, judgments) Test Flow is still not detailed enough to get a full picture without reproducing Test Case manually. Another reason could be a lack of implementation. Since logical judgments and validations are not implemented they’re not reported. And automation developer still must take care of making Execution Report Test Case oriented.

Test Report

Embedded from Test Tool. Contains detailed information on GUI and Logical (Verification/Validation) steps performed providing end-user with the fully reproducible Test Flow. Optionally may contain Fail reports and other reconciliation details.
This is something looking like as it’s done manually but better (because manual tester will get really bored accurately documenting hundreds of Test Cases executed).

 Operation
- Do you want to have special stuff executing Automated Test Cases or you need a “click-and-start” procedure?

Developer Only

Only automation developers can setup and start scripts.
When there is no convenient “front-end”, and all the configurations are part of the code, only the person very familiar with the script can successfully launch it.

User Friendly

Only basic computer skills required to start Automated Test Plan. Short training required for the unified setup procedure.
HP/Mercury Quality Center is one of the perfect examples.
However, QC doesn’t provide much customization features automatically applicable to the batch you’re starting. So if it’s something more complicated than a set of independent Test Cases executing on a single environment, advanced design and implementation of an Automated Test Plan document is required. But custom development must still strictly follow easy-and-simple-to-use requirements.

Design
- When it comes to using and re-using automated test assets, is it possible for automation developers only, or it’s built convenient enough to be successfully used by non-technical testers without big exposure to automation?

 Developer Only

Only developers can create Test Plans or do modification of existing ones; due to hard-codings everything requires code change.

Specialist Only

Specially trained QA with technical knowledge can do minor modification. Due to all the test logic stored as code development the risk is still high.
Half-way to reach a convenient customization is passed but there is a lot of things still done manually.

User Friendly

Non-technical user (QA, BA) can create new and modify existing test flow using Keyword-Driven front-end.
It could be as simple as editing Excel spreadsheet, or drag-n-drop operations on a visual call tree.
HP Business Process Testing Framework is a nice example of that.

Swiftness
- How quickly the existing Test Suite allows adding (development) new test assets, like Test Cases and reusable testing functionalities?

Individual Creation

Each manual Test Case requires development (or recording) of Test Script. No framework.

Modular Programming

Test Cases developed individually; common functions are maintained in library files. Sub-framework level.

Visual Prototyping

Test Case automation DOES NOT require programming. Framework fully supports Business/Test Logic execution from the back-end.
Hybrid Keyword/Data Driven Framework implementation is required as an initial investment.
Or you can try some of the commercial ones:
HP Business Process Testing (VBScript is a programming language)
Worksoft Certify (doesn’t have programming language support).

Original date: 14 Apr 2009, 1:30pm

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness 

Usability Requirements Matrix 1

Applicability
- How would you want to use Automated Testing solution?
Basic Regression Testing

Implemented as a sequence of data entry steps going through the screens.
No or simplified verification based on hard-coded checkpoints.

Comprehensive Functional/Regression Testing

Thorough and robust automated testing. Test script is going through the screens, verifying functionalities, validating test results.
End-user can see high-level picture on the Test Report and make final judgment quickly. In case of need, though, Test Report provides all the details down to basic testing steps performed so that end-user can get the test flow reproduced easily.

All-phase execution (DEV, QA, UAT)

Suitable for Smoke, Integration, Functional/Regression, Sanity, Functional/UAT testing.
Test results delivery to developers, QA, BA, and Client-side.
While being comprehensive Functional/Regression Testing solution (with the same requirements as above), it needs to be implemented in a user-friendly and generic way, so that getting used to requires small learning curve; easy for non-technical user; specific for technical user.
Test Report must be transferable and embedded from the Testing Tool.
Dataset is flexible, easily maintainable and customizable.
 

 Utilization
- Do you want to utilize the scripts for setup and configuration procedures, in addition to testing needs?
Automated Preload

Data entry procedures providing environment with data to perform testing. Could be also used for database testing, volume testing, and release comparison testing.
I also had experience successfully using automated data entry flow in Production for data migration type projects. However, this requires perfectly robust scripts, extremely careful planning through, and cooperation with other tech teams (to do a back-up, provide special access, disable certain production features, etc.).

Automated Configuration

All the setup and preparation procedures, like setting required configuration, creating/modifying options, etc.
Being properly built saves a lot of manual time, and gives a turnover advantage.
 

 Coverage
- How thorough you want your automated testing to be? What kinds of testing you want to automate?
Basic Data Entry Flow

Automated data entry Test Case. Populating mandatory fields only. PASS condition: reach the last screen.
Similar to Sanity Testing performed manually.

Full Data Entry Flow

Automated data entry Test Case. Populating all fields (based on input data provided). PASS condition: reach the last screen.
Similar to simplified positive functional testing performed manually.

Smoke Test Flow

Same as Full Data Entry Flow + all errors handled and reported so that gives an unbreakable execution with high coverage advantage.

One of the very beneficial ways to use would be:
• prepare automated smoke/regression test plans;
• let developers launch them after the build / overnight deployment;
• have test results available for PM, QA, and Dev next morning.

Developers can catch and fix most obvious defects on-the-fly, without getting QA involved.
QA have more time to focus on failure analysis, and more time to spend on a manual complicated / ad-hock testing.

Test Flow Validation

Same as Smoke Test Flow + automated validation, PASS/FAIL judgment along with Severity/Consistency report.
More smart type of automated testing with validation (judgments) of fails implemented. Helps to quickly identify known defects, or create defect reports for the newly discovered ones.
Gives better time-saving advantage since manual validation operations are not necessary.

Regression Test Flow

Same as Test Flow Validation + on-screen validations (application response, contents of dropdowns, presence or absence of dynamic GUI, etc.).
While keeping robustness and validations features, provides additional testing functionalities (not directly related to the business logic flow) for Functional/Regression testing needs.
However, building additional verification & validation requires stronger business logic support by Subject Matter Expert, and extra data build-up / maintenance efforts.

Report Validation

Query system by submitting user requests through the front-end and validate reports.
“Gray Box” testing of stored procedures implementing SQL queries could be very effectively covered with automated testing scripts. In addition, queries may cover boundary and overflow conditions.

Positive/Negative Boundary Testing and Data Acceptance

Test application’s ability to handle value at borders of the allowed range, and outside of it. Check for overflows and crashes.
As Negative Testing, validate application’s response on intentionally invalid data input, or wrong combinations of data, or user actions.
Complexity and coverage of this automated testing type is nearly as thorough as a manual ad-hock testing. However, reasonable limit of test flows to automate should be set (risk assessment based on impact), to avoid code maintenance issues.
 

Reliability
- How reliable test results must be? Can testers trust test results even without manually reproducing the case or failure only means that “something went wrong” and needs to be investigated manually?
Full manual investigation

Test results require manual processing in order to find out if occurred FAILs were the actual defects in the application. Manual processing means manual re-testing of Test Case failed in order to find a reason of failure.

Final human judgment

The each fail occurred is reported in detail, showing up the application state and level of impact. No need to reproduce Test Case manually.
Known fails are analyzed and reported by the script.
Root cause analysis is performed by human.
 

Volume
- How many automated Test Cases you want to use and how would you want it to be organized?
Execute single TC

Each Test Case script started manually.
This is still more manual rather than automated testing.

Execute hard-coded Test Scenario

According to Test Plan, developer creates batch of Test Case scripts. Batch file performs execution calls in a defined sequence, Test Case scripts used “as is”. That means, individual configuration/customization of Test Case scripts mist be performed prior to the batch execution.
In case of multi-environment or multi-dataset test requirements batch execution is less convenient due to manual maintenance efforts required.

Execute customizable Test Plan

List/sequence of Test Cases with customizable Test Data Input, Application Release, Benchmarks, Reporting, etc.
End-user (BA, QA) can build / modify Test Plans using previously automated Test Cases available; no programming is required.
Test Driver script performs automatic analysis and invocation of Test Cases scheduled, passing user-defined high-level input, like Application / URL to use, and Test Data set. Using customizable keys Tester can set Test Plan benchmarks and control automated test flow at the Test Plan level.
Built and baselined Test Plans can be successfully used by Dev, BA, and non-technical Jr.QA staff.

Overview of Usability requirements will be continued with Transparency, Operation, Design, and Swiftness.

QA Automation – GUI Function Wrapping

Some of the statements below will be covered in detail in my subsequent posts. Now I use them to get straight to Test Step requirements.
Please read details below the picture.

Quality Assurance Functional/Regression Automated Testing and Test Automation Requirements: Usability, Maintainability, Scalability, and Robustness.

Let’s break them down now for implementation of a single GUI Step.

Usability / Coverage

Test Step encapsulates verification of GUI Object and Test Data.

GUI Object verification includes.

Accessibility
• If the Object is defined by logical name, its description is available in Repository.
• If the Object is defined by description, Object’s Parent can return Object Instance.
• Object Instance is not empty ( != NULL).

Availability
• The Object must EXIST on the GUI (and this is enough for the Property Read operation).
• For input-type operation the Object must be ENABLED.
• For Screen Read operation the Object must be DISPLAYED.

Test Data verification includes acceptability by an Object and DOES NOT include any business validation. If Application Under Test for whatever the reason didn’t like the input, it brings error message or other response, that is captured, (optionally) analyzed, and reported by the automated testing system.

Acceptability

• Number of arguments provided must match number of arguments that Object’s method can accept.
Example. To select a cell on a table both row and column must be provided.
• Type of arguments to pass in matches type of arguments Object’s method can accept.
Example. To select a cell on a table row number and column name must be provided.
• Type of supplied data matches types that Object’s method can accept
Example. A CheckBox accepts only 3 values that define possible states of it: ON, OFF, or DIMMED

Robustness

Synchronization, error and exception handling provide stable and unbreakable test flow.

Synchronization
• If specified, the Test Step Function should wait for the Object reaching defined state (E.g. EXISTS, ENABLED, DISPLAYED) within default or defined timeframe.

Error handling
• In case of failed operation caused by any matter verification code handles the error, and provides recovery and negative reporting.
• In case of unexpected AUT response (I.e. Pop-up Dialog) exception handler catches it, handles, and passes control back to the targeted GUI Parent form.

Usability / Investigation and Reporting

Test Flow reporting provides end-user with the complete and detailed information, and that allows reproducing even without manually going through the Test Case. Reporting of Object and AUT status description, Test Data log, and judgments made serves that purpose.
In case of failures investigation should be performed, by capturing state of application-under-test, GUI form and GUI object, isolating and reporting properties of the mentioned components.

Description
• In case of Negative Reporting, description provides the reason of the FAIL.
Example 1. Failed to input value. Object is not enabled.
Example 2. Failed to select list item. Item does not exist.
• In case of Positive Reporting, description provides details about the operation performed.
Example 1. Pressed button.
Example 2. Retrieved object data.

Data log
• Data used in input, retrieval or verification are provided to the end-user.

Judgments
• Describe PASS condition used.
• Describe FAIL condition used.
• Describe Cut-Off condition (timeout, boundaries, etc.) used.

Finally, you can see below an example of the Actual Test Report generated after the execution of several GUI steps (business data scrambled).

Original date:  4 Dec 2008, 1:49pm

Test Case Automation Cycle

What is it?

IDEF0 is abbreviation for Integration Definitions for Functional Modeling [of Systems].
Zero means type of modeling. (IDEF0 – function modeling, IDEF1 – information modeling, IDEF2 – dynamics modeling).

How it looks?

Image courtesy: Wikipedia

How it works?

Image courtesy: Berry College

2-tier Data-Driven Test Automation Architecture (WinRunner) – Advanced Reporting

Click to see full-size in a separate window