Assumption versus Investigation

I recently replied to Ben Kelly’s post “Adventures in GUI Automation” and he got interested in using QTP’s Object Repositories dynamically in functions, instead of attaching them to Actions. He guessed about performance boost as an advantage of such an approach and I agreed without much of a doubt. (I have to say, I use OR’s dynamically in my hybrid framework for other reasons: flexibility, maintainability, portability…). However, GUI interaction steps must be performed in synchronization with the Application-Under-Test (AUT), and testing scripts have to wait being idle while AUT reloading web pages or switching desktop windows – so scripts’ performance is never an issue.

Nevertheless, I still felt a little anxious since I didn’t really test performance boundaries. So finally I took some time, prepared and run tests – to get really surprising results!

Application Under Test

Microsoft Windows (R) Calculator was used as a sample AUT.

I used QTP’s Object Repository Manager to create an embedded file. The file contained about 30 entries (Test Object Definitions).

I also created ORTest2.tsr file by duplicating “Calculator” window in it. The second file contained doubled number of objects’ descriptions. Note that file size wasn’t doubled (from 294,912 bytes to 360,448 bytes).

Performance Test #1: Add/Remove Repository

I used the following script to conduct a sample test.

Set objTimer = MercuryTimers("RCTest")
MercuryTimers("RCTest").Start
 
For Iter = 1 to 100
 intPos = RepositoriesCollection.Find("C:\Temp\ORTest.tsr")
 If intPos = -1 Then
  RepositoriesCollection.Add "C:\Temp\ORTest.tsr"
 End If
 intPos = RepositoriesCollection.Find("C:\Temp\ORTest.tsr")
 If intPos <> -1 Then
  RepositoriesCollection.Remove intPos
 End If
Next

intTimeElapsed = CInt(objTimer.ElapsedTime/1000)
dblTimeElapsed = objTimer.ElapsedTime/1000

MsgBox(intTimeElapsed)

Note that Add/Remove block is repeated 100 of times to average the results.

The script was executed 6 times (3 times per Object Repository file) to average the results.

So, doubling the number of Object Descriptions affected performance by increasing round time by 0.745 sec, or a single loop time by 0.00745 sec.

As we can see from this test, the absolute time of an operation do not change significantly. Adding that loading/unloading of Object Repositories occurs during GUI scene change when script does not interact with an AUT anyway, we may conclude that performance boost is not a critical advantage.

Performance Test #2: Object Repository versus Descriptive Programming

I often hear from many automation developers that they don’t use Object Repositories because “it’s much slower than Descriptive Programming”, and because “you can’t use them with modular data-driven framework”.

Modular framework concept assumes building a set of testing libraries and calling high-level test logic functions from the main action. Since Object Repositories can be attached to QTP Actions only, that means all possibly required OR’s have to be loaded at once in the beginning of test and stay loaded until the end – and here comes the slowdown!

In the same time, Descriptive Programming way to address GUI objects means hard-coding hundreds (or THOUSANDS) of technical, business-meaningless values (which may change from build to build) all the way across the scripts. Any person, who will have to work with the scripts heavily using hard-coded values, will surely damn those maintenance bombs.

The last part that we need to investigate is how same code using Descriptive Programming to address GUI objects performs versus the code using Object Repository for mapping GUI objects’ names.

ORTest Action

“ORTest” (- Object Repository Test) Action has “ORTest.tsr” file attached to it. The main calling Action doesn’t have any shared OR’s attached.

Window("Calculator").Activate
Window("Calculator").WinButton("7").Click
Window("Calculator").WinButton("8").Click
Window("Calculator").WinButton("9").Click
Window("Calculator").WinButton("/").Click
Window("Calculator").WinButton("sqt").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("4").Click
Window("Calculator").WinButton("5").Click
Window("Calculator").WinButton("6").Click
Window("Calculator").WinButton("*").Click
Window("Calculator").WinButton("%").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("1").Click
Window("Calculator").WinButton("2").Click
Window("Calculator").WinButton("3").Click
Window("Calculator").WinButton("-").Click
Window("Calculator").WinButton("1/x").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("0").Click
Window("Calculator").WinButton("+/-").Click
Window("Calculator").WinButton(".").Click
Window("Calculator").WinButton("+").Click
Window("Calculator").WinButton("=").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("M+").Click
Window("Calculator").WinButton("MS").Click
Window("Calculator").WinButton("MR").Click
Window("Calculator").WinButton("MC").Click
Window("Calculator").WinButton("Backspace").Click
Window("Calculator").WinButton("CE").Click
Window("Calculator").WinButton("C").Click

As you can see, it’s just a sequence of clicking operations with a purpose of addressing GUI objects from Object Repository.

DPTest Action

“DPTest” (- Descriptive Programming Test) Action has no shared OR’s attached and executed independently from the first Action (in a separate script runs).

Window("text:=Calculator").Activate
Window("text:=Calculator").WinButton("text:=7").Click
Window("text:=Calculator").WinButton("text:=8").Click
Window("text:=Calculator").WinButton("text:=9").Click
Window("text:=Calculator").WinButton("text:=/").Click
Window("text:=Calculator").WinButton("text:=sqt").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=4").Click
Window("text:=Calculator").WinButton("text:=5").Click
Window("text:=Calculator").WinButton("text:=6").Click
Window("text:=Calculator").WinButton("text:=\*").Click
Window("text:=Calculator").WinButton("text:=%").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=1").Click
Window("text:=Calculator").WinButton("text:=2").Click
Window("text:=Calculator").WinButton("text:=3").Click
Window("text:=Calculator").WinButton("text:=-").Click
Window("text:=Calculator").WinButton("text:=1/x").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=0").Click
Window("text:=Calculator").WinButton("text:=\+/-").Click
Window("text:=Calculator").WinButton("text:=\.").Click
Window("text:=Calculator").WinButton("text:=\+").Click
Window("text:=Calculator").WinButton("text:==").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=M\+").Click
Window("text:=Calculator").WinButton("text:=MS").Click
Window("text:=Calculator").WinButton("text:=MR").Click
Window("text:=Calculator").WinButton("text:=MC").Click
Window("text:=Calculator").WinButton("text:=Backspace").Click
Window("text:=Calculator").WinButton("text:=CE").Click
Window("text:=Calculator").WinButton("text:=C").Click

Note that Object Description property value always is a Regular Expression hence backslashes in descriptions of certain buttons.

Main Action

Main Action code is used to measure performance in a way similar to the first test.

Set objTimer = MercuryTimers("ORTest")
MercuryTimers("ORTest").Start
 
For Iter = 1 to 10
 RunAction "ORTest", oneIteration
' RunAction "DPTest", oneIteration
Next

intTimeElapsed = CInt(objTimer.ElapsedTime/1000)
dblTimeElapsed = objTimer.ElapsedTime/1000

MsgBox(intTimeElapsed)

Note that each Action is called 10 times to average the results.

There were 10 separate executions of the script (5 times per Action) to average the results.

Here comes the surprise!

As we can see from test results, same code using Descriptive Programming is executed significantly slower! In average, round time (10 loops) is increased by 15.331 sec, and a single loop time is increased by ~1.5 sec. We can also see that deviation between results is greater for DPTest Action.

Where is the main difference?

Physical Descriptions of an object are represented as a string (e.g. “text:=\+/-“). Each value must be found and extracted, and each value is treated as a Regular Expression which is matched against properties of Run-time GUI object. String operations always take much more time to execute.

Conclusion

I do use both object mapping to Repositories and addressing directly through Descriptions. I’m not really concerned about performance of scripts which must follow performance of Application-Under-Test. I care about Usability, Maintainability, Scalability, and Robustness.

In a context of dynamic GUI recognition Descriptive Programming is much better and it’s much more convenient than maintaining multiple entries in Object Repository.

In a context of pre-defined GUI using business-meaningful names mapped to separately maintainable entries in shared Object Repositories is much wiser than infecting your scripts with hundreds of hard-coded values.