Is your automation asset or debt?

Test Assets?

Vendors, offering Test Automation tools, often refer to automation and its elements (i.e. scripts, data files, logs) as “test assets”. I could certainly agree with such reference, if it’s based on the assumption that since time, money, and effort were put [invested] to create those elements, they are assets. But is anything you pay for worth its money? Is anything you spent time creating actually valuable?

To answer a question about Test Automation value we need to answer a question about value of Testing.

Value of Testing

Testing itself does not create a product. Testing is an internal (or even external) informational service.
It might help making shipment decisions and it might help avoiding costly damages caused by issues in a software product.

However, possible damages could be minimal and other risks might be low enough to be not concerned about testing.

Obviously, in such cases Test Automation may create no value.

A Need for Automation

Let’s assume the need for automation has been established as a “Smoke Test Suite that is supposed to be used as a Build Acceptance Test”.

Let’s assume that the build consists of a 100 of Test Cases that could be executed automatically in 4 hours, and manually in 2 days.

Downsides

Build Acceptance Test is just a preface to cycles of testing. If any major issues were detected by the test, the build is rejected without opening defect reports. Hence the minor value of Build Acceptance Test itself – we could only talk about saving human time.

Hence, roughly a possible value could be defined as (16 – 4 = 12) hours.
But let’s now take in account the following expenses incurred:

1) Initial investment

• Number of hours spent creating 100 of automated Test Cases
• Number of hours spent preparing Test Data templates for the scripts

2) Re-occurring expenses

• Number of hours spent updating scripts (i.e. adjusting to changes in the logic and GUI in the test flow)
• Number of hours spent updating test data
• Time taken to setup Test Environment (steps, required for testing scripts)
• Time taken to evaluate test execution results

To make task even simpler, we will not count expenses associated with the initial creation of scripts.

Let’s put some numbers for re-occurring expenses.

Updating scripts

10 minutes per script -> 16 hours. Where is the value, again? If testing manually, you would be done by the time you are not even ready to start your scripts! And don’t forget, while testing manually, you can reject a build any moment you encounter a serious issue.

Updating test data / setting up Test Environment

While preparing a Test Environment or updating Test Data, an experienced tester can instantly spot issues, spend little time investigating, and reject the build. Again, automation doesn’t even have a chance to start.

Focus on the real asset

Having code or data repository is important.
This way it saves [reduces] time and effort people spend creating and maintaining them.

But neither testing scripts nor test data create value. The only real and main Test Assets on the project are People – software testing professionals. Some of them need to be proficient in automation, while the whole intent should stay focused on Testing.