preload

A little QTP performance test: Object Repository vs. Descriptive Programming

Posted by Albert Gareev on Mar 18, 2010 | Categories: Source codeTools

Assumption versus Investigation

I recently replied to Ben Kelly’s post “Adventures in GUI Automation” and he got interested in using QTP’s Object Repositories dynamically in functions, instead of attaching them to Actions. He guessed about performance boost as an advantage of such an approach and I agreed without much of a doubt. (I have to say, I use OR’s dynamically in my hybrid framework for other reasons: flexibility, maintainability, portability…). However, GUI interaction steps must be performed in synchronization with the Application-Under-Test (AUT), and testing scripts have to wait being idle while AUT reloading web pages or switching desktop windows – so scripts’ performance is never an issue.

Nevertheless, I still felt a little anxious since I didn’t really test performance boundaries. So finally I took some time, prepared and run tests – to get really surprising results!

Application Under Test

 

Calculator

Microsoft Windows (R) Calculator was used as a sample AUT.

I used QTP’s Object Repository Manager to create an embedded file. The file contained about 30 entries (Test Object Definitions).

I also created ORTest2.tsr file by duplicating “Calculator” window in it. The second file contained doubled number of objects’ descriptions. Note that file size wasn’t doubled (from 294,912 bytes to 360,448 bytes).

Performance Test #1: Add/Remove Repository

I used the following script to conduct a sample test.


Set objTimer = MercuryTimers("RCTest")
MercuryTimers("RCTest").Start
 
For Iter = 1 to 100
 intPos = RepositoriesCollection.Find("C:\Temp\ORTest.tsr")
 If intPos = -1 Then
  RepositoriesCollection.Add "C:\Temp\ORTest.tsr"
 End If
 intPos = RepositoriesCollection.Find("C:\Temp\ORTest.tsr")
 If intPos <> -1 Then
  RepositoriesCollection.Remove intPos
 End If
Next

intTimeElapsed = CInt(objTimer.ElapsedTime/1000)
dblTimeElapsed = objTimer.ElapsedTime/1000

MsgBox(intTimeElapsed)

Note that Add/Remove block is repeated 100 of times to average the results.

The script was executed 6 times (3 times per Object Repository file) to average the results.

  Round 1 Round 2 Round 3 Average
ORTest.TSR 18.609 18.687 17.937 18.411
ORTest2.TSR 18.984 19.562 18.921 19.15567
Diff 0.375 0.875 0.984 0.744667

 

So, doubling the number of Object Descriptions affected performance by increasing round time by 0.745 sec, or a single loop time by 0.00745 sec.

As we can see from this test, the absolute time of an operation do not change significantly. Adding that loading/unloading of Object Repositories occurs during GUI scene change when script does not interact with an AUT anyway, we may conclude that performance boost is not a critical advantage.

Performance Test #2: Object Repository versus Descriptive Programming

I often hear from many automation developers that they don’t use Object Repositories because “it’s much slower than Descriptive Programming”, and because “you can’t use them with modular data-driven framework”.

Modular framework concept assumes building a set of testing libraries and calling high-level test logic functions from the main action. Since Object Repositories can be attached to QTP Actions only, that means all possibly required OR’s have to be loaded at once in the beginning of test and stay loaded until the end – and here comes the slowdown!

In the same time, Descriptive Programming way to address GUI objects means hard-coding hundreds (or THOUSANDS) of technical, business-meaningless values (which may change from build to build) all the way across the scripts. Any person, who will have to work with the scripts heavily using hard-coded values, will surely damn those maintenance bombs.

The last part that we need to investigate is how same code using Descriptive Programming to address GUI objects performs versus the code using Object Repository for mapping GUI objects’ names.

ORTest Action

“ORTest” (- Object Repository Test) Action has “ORTest.tsr” file attached to it. The main calling Action doesn’t have any shared OR’s attached.


Window("Calculator").Activate
Window("Calculator").WinButton("7").Click
Window("Calculator").WinButton("8").Click
Window("Calculator").WinButton("9").Click
Window("Calculator").WinButton("/").Click
Window("Calculator").WinButton("sqt").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("4").Click
Window("Calculator").WinButton("5").Click
Window("Calculator").WinButton("6").Click
Window("Calculator").WinButton("*").Click
Window("Calculator").WinButton("%").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("1").Click
Window("Calculator").WinButton("2").Click
Window("Calculator").WinButton("3").Click
Window("Calculator").WinButton("-").Click
Window("Calculator").WinButton("1/x").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("0").Click
Window("Calculator").WinButton("+/-").Click
Window("Calculator").WinButton(".").Click
Window("Calculator").WinButton("+").Click
Window("Calculator").WinButton("=").Click
Window("Calculator").WinButton("C").Click
Window("Calculator").WinButton("M+").Click
Window("Calculator").WinButton("MS").Click
Window("Calculator").WinButton("MR").Click
Window("Calculator").WinButton("MC").Click
Window("Calculator").WinButton("Backspace").Click
Window("Calculator").WinButton("CE").Click
Window("Calculator").WinButton("C").Click

As you can see, it’s just a sequence of clicking operations with a purpose of addressing GUI objects from Object Repository.

DPTest Action

“DPTest” (- Descriptive Programming Test) Action has no shared OR’s attached and executed independently from the first Action (in a separate script runs).


Window("text:=Calculator").Activate
Window("text:=Calculator").WinButton("text:=7").Click
Window("text:=Calculator").WinButton("text:=8").Click
Window("text:=Calculator").WinButton("text:=9").Click
Window("text:=Calculator").WinButton("text:=/").Click
Window("text:=Calculator").WinButton("text:=sqt").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=4").Click
Window("text:=Calculator").WinButton("text:=5").Click
Window("text:=Calculator").WinButton("text:=6").Click
Window("text:=Calculator").WinButton("text:=\*").Click
Window("text:=Calculator").WinButton("text:=%").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=1").Click
Window("text:=Calculator").WinButton("text:=2").Click
Window("text:=Calculator").WinButton("text:=3").Click
Window("text:=Calculator").WinButton("text:=-").Click
Window("text:=Calculator").WinButton("text:=1/x").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=0").Click
Window("text:=Calculator").WinButton("text:=\+/-").Click
Window("text:=Calculator").WinButton("text:=\.").Click
Window("text:=Calculator").WinButton("text:=\+").Click
Window("text:=Calculator").WinButton("text:==").Click
Window("text:=Calculator").WinButton("text:=C").Click
Window("text:=Calculator").WinButton("text:=M\+").Click
Window("text:=Calculator").WinButton("text:=MS").Click
Window("text:=Calculator").WinButton("text:=MR").Click
Window("text:=Calculator").WinButton("text:=MC").Click
Window("text:=Calculator").WinButton("text:=Backspace").Click
Window("text:=Calculator").WinButton("text:=CE").Click
Window("text:=Calculator").WinButton("text:=C").Click

Note that Object Description property value always is a Regular Expression hence backslashes in descriptions of certain buttons.

Main Action

Main Action code is used to measure performance in a way similar to the first test.


Set objTimer = MercuryTimers("ORTest")
MercuryTimers("ORTest").Start
 
For Iter = 1 to 10
 RunAction "ORTest", oneIteration
' RunAction "DPTest", oneIteration
Next

intTimeElapsed = CInt(objTimer.ElapsedTime/1000)
dblTimeElapsed = objTimer.ElapsedTime/1000

MsgBox(intTimeElapsed)

Note that each Action is called 10 times to average the results.

There were 10 separate executions of the script (5 times per Action) to average the results.

  Round 1 Round 2 Round 3 Round 4 Round 5 Average
ORTest 28.516 26.719 26.212 29.178 25.515 27.228
DPTest 45.141 47.514 36.562 37.734 45.843 42.5588
Diff 16.625 20.795 10.35 8.556 20.328 15.3308

 

Here comes the surprise!

As we can see from test results, same code using Descriptive Programming is executed significantly slower! In average, round time (10 loops) is increased by 15.331 sec, and a single loop time is increased by ~1.5 sec. We can also see that deviation between results is greater for DPTest Action.

Where is the main difference?

Physical Descriptions of an object are represented as a string (e.g. “text:=\+/-“). Each value must be found and extracted, and each value is treated as a Regular Expression which is matched against properties of Run-time GUI object. String operations always take much more time to execute.

Conclusion

I do use both object mapping to Repositories and addressing directly through Descriptions. I’m not really concerned about performance of scripts which must follow performance of Application-Under-Test. I care about Usability, Maintainability, Scalability, and Robustness.

In a context of dynamic GUI recognition Descriptive Programming is much better and it’s much more convenient than maintaining multiple entries in Object Repository.

In a context of pre-defined GUI using business-meaningful names mapped to separately maintainable entries in shared Object Repositories is much wiser than infecting your scripts with hundreds of hard-coded values.


  • 3 responses to "A little QTP performance test: Object Repository vs. Descriptive Programming"

  • Jim
    22nd March 2010 at 15:07

    Hi Albert,

    If I decided to go with QTP Actions and not functions what is the best way to pass parameters while calling them?

    Thanks,
    Jim

    [ Albert’s Reply.

    Hi Jim,
    You can use Parameter built-in object for primary arguments, Environment built-in object for configuration data, and Datatable built-in object for Test Case data. ]

  • Nathan
    18th April 2010 at 21:30

    Wow. I never wanted to use so much hardcodings in QTP scripts. Now I’m armed to argue about this question at work.

  • GorillaTester
    13th February 2014 at 13:30

    Your test is interesting, but the performance characteristics of the object repository are not linear as the number of objects grows,
    [Albert’s reply: care to back it up?]
    whereas they are linear when using descriptive programming.
    [Albert’s reply: care to back it up?]
    Try the test with 10,000 objects in your repository. The performance will slow using the OR, while the speed of the descriptive programming remains constant.
    [Albert’s reply. Why don’t YOU make the test, and publish it?
    I’m also curious to find out about maintenance effort for 10,000 descriptions scattered across the script. Not mentioning that having a repository with even 1,000 objects would be a rookie design mistake.]

Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported
This work by Albert Gareev is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported.