Publication URL:  http://www.qaguild.com/weekly_archives.php?UID=55

Data-Driven != Parameterized

(Or How to Avoid Test Flow Hard-Coding And Dramatically Reduce Code Maintenance Cost in Functional Test Automation)

 Many times looking at the busy screen together with Business Analyst I used to discuss dozens of manual Test Cases they execute to cover the functionalities of this single Window or Dialog. Business Logic seemed to be quite similar but different enough to produce a multiple Test Logic flows.

So the dilemma I had was about either to follow manual test requirements and create multiple scripts setting maintenance trap or to limit number of Test Cases to automate but reduce testing coverage as a result. Both ways were still accepted as reasonable by QA Manager but reducing coverage was really against my QA nature.

Looking at the screen alone I was storming the simple concept: the Application still has just a single screen providing all the functionalities. If one program can do it why another program requires multiple implementations? While being different on steps to perform the all Test Cases have the same entry point and major PASS/FAIL condition.

The solution I came to I’m going to demonstrate using MS Windows Calculator as the example.

Manual Test Cases

1. A + B = C

Test Data and Steps.

Input. A = 2; B = 3.

Expected result. C = 5.

2. A – B = C

Test Data and Steps.

Input. A = 2; B = 3.

Expected result. C = -1.

3. A * B = C

Test Data and Steps.

Input. A = 2; B = 3.

Expected result. C = 6.

4. A / B = C

Test Data and Steps.

Input. A = 6; B = 3.

Expected result. C = 2.

Recorded Scripts (Hard-coded Data and Logic)

 TC1.

Click “2”;Click “+”;Click “3”;Click “=”.

Read value C.

Verify C is equal 5.

 

TC2.

Click “2”;Click “-”;Click “3”;Click “=”.

Read value C.

Verify C is equal -1.

 

TC3.

Click “2”;Click “*”;Click “3”;Click “=”.

Read value C.

Verify C is equal 6.

 

TC4.

Click “6”;Click “/”;Click “3”;Click “=”.

Read value C.

Verify C is equal 2.

Parameterized Scripts (Parameterized Data / Hard-coded Logic)

 Reusable Functions

 

Reusable Function 1: Input Number.

Description: Clicks on number buttons in accordance to the number passed-in. No return value.

Reusable Function 2: Compare Number.

Description: Compare 2 passed-in numbers. Return TRUE if equal FALSE otherwise.

 Scripts

 

TC1.

Input Number A;

Click “+”;

Input Number B;

Click “=”.

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

 

TC2.

Input Number A;

Click “-”;

Input Number B;

Click “=”;

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

 

TC3.

Input Number A;

Click “*”;

Input Number B;

Click “=”;

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

 

TC4.

Input Number A;

Click “/”;

Input Number B;

Click “=”;

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

Data Table

Test Case	A	B	C-Expected
TC1	2	3	5
TC2	2	3	-1
TC3	2	3	6
TC4	6	3	2

 

Analysis

 
Parameterization of the values obviously extends Test Coverage within the same Test Case group. Whether it’s 2 + 3 or 200 + 300 same script would cover all the combination provided.

However the logic is still hard-coded as a sequence of steps to perform.

Code duplication rate for the scripts exceeds 75%.

Data-Driven Script (Data as Input and Test Flow control)

 Reusable Functions

 

Reusable Function 3: Input Operator.

Description: Clicks on the operator button in accordance to the operator passed-in. Skips if the argument is empty. No return value.

 Script

 

Input Number A;

Input Operator O1; Input Operator O2; Input Operator O3; Input Operator O4;

Input Number B;

Click “=”.

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

 Data Table

Test Case	A	O1	O2	O3	O4	B	C-Expected
TC1	2	+				3	5
TC2	2		–			3	-1
TC3	2			*		3	6
TC4	6				/	3	2

Analysis

Four separate scripts were converted to a single one.
Amount of the code was decreased to 37.5 % of the initial one without losing the initial coverage.

Manual Test Cases (Extended Coverage)

 Maybe it was initially planned or due to a Production incident Test Coverage was requested to be extended with the following Test Cases.

 

A1 + A2 + A3 + … + An = C

 

A1 – B2 + A3 – B4 + … + An – … – Bn = C

 

Iterative Data-Driven Script

 Reusable Functions

 Reusable Function 4: Get Operation

Description: Based on the parameter passed-in (Operation Name) returns the following record:

O1

O2

O3

O4

B

 Script

Input Number A0;

Begin loop through comma-separated list of Operations

    Get Operation;

    Input Operator O1; Input Operator O2;

    Input Operator O3; Input Operator O4;

    Input Number B;

End Loop If no more Operations in the list

Click “=”.

Read value as C-Actual;

Compare Number C-Actual, C-Expected.

Primary Data Table

Test Case	A0	Operations	C-Expected
TC1.1	2	ADD.01	5
TC1.2	2	ADD.02	3
TC2.1	2	SUB.01	-1
TC2.2	2	SUB.02	-3
TC3.1	2	MUL.01	6
TC3.2	2	MUL.02	8
TC4.1	6	DIV.01	2
TC4.2	6	DIV.02	1
TC5.1	2	ADD.01, ADD.02	6
TC6.1	2	ADD.01, SUB.01	2
TC7.1	2	ADD.01, ADD.02,SUB.02	1
TC8.1	2	ADD.01, MUL.01,SUB.01	3
TC9.1	2	MUL.01,SUB.01, ADD.02,	4

 Secondary Data Table

Operations	O1	O2	O3	O4	B
ADD.01	+				3
ADD.02	+				1
SUB.01		–			3
SUB.02		–			5
MUL.01			*		3
MUL.02			*		4
DIV.01				/	3
DIV.02				/	6

Analysis

Single script covers the required Addition/Subtraction Test Cases and provides coverage for the all possible combinations of Addition/Subtraction/Multiplication/Division.

3 extra lines of the code were added to the script.

Development of the additional reusable function was required.

Within the same type/class of Test Cases Test Logic creation/modification does not require code change and could be performed by any user with basic Excel skills.

Conclusion

In fact, what single Business Program does could be covered with the single Testing Program while manual testing requires a set of Test Cases. Utilizing a simple trick “skip step if no data” testing script produces test flow combinations covering different manual testing sequences.

There are still mandatory prerequisites: common GUI, same entry point, same major PASS/FAIL conditions.

The code becomes extremely compact and that saves a lot of time on the maintenance phase. This approach also reduces data maintenance efforts as tester may leave empty any optional inputs, as well as reuse secondary data rows. The coverage is getting extended even beyond the required level.

P.S. The real Business Window I had to cover contained about 50 Input fields on 3 tabs. With 2 independent loops, same fields could be populated undefined (from 1 to 100) number of times. Submitted within a loop data were displayed on the on-screen table as records. The Application responded to an input dynamically showing/hiding and enabling/disabling co-dependant Input and Message fields, as well as changing default values and contents of dropdown-lists.

The total number of Manual Test Cases exceeding 100 was successfully covered with a single Iterative Data-Driven script.