This is another my notebook entry, but entry unusual. I started it while playing chess.

This is also one of the proud parent moments: my 9 years old son brought this initiative of evening chess parties – all by himself!
As much as we, parents, try to protect and guide our children, we also try to develop them self-driven and self-sustainable – and that’s not easy, you know.

A couple of years ago my dad introduced chess to my son, and the three of us had some quality time learning about this game. But it didn’t stick that time.
Then this winter my son asked me to help installing chess game on his tablet, and I saw him playing it now and then, along with arcades, shooters, and other typical for his age games.
And suddenly my son approaches me, saying that he is one of the best in the school’s chess club, and he wants to get even better by playing regularly with me. He asks me to dedicate time for regular evening sessions and promises to help more with household stuff to help freeing up time.
Well, that’s exactly the moment of feeling proud of your child, and having real confirmation that you’re doing a good job as a parent.

More to say, it’s been over a month since we started playing regularly, lots of losses for my son, and he is still persistent and strong spirited about the game.

Now, here are the connections with testing.

Have a strategy, and be ready  to change it

It is about goals, not “steps to perform”

Winning the party is an ultimate goal, which is reached by series of small wins (mixed with some losses, possibly) through the party. No one can prescribe a sequence of moves to win the game.

Know the context

You need to know what kind of game you play: modern chess, hexagonal chess, xiangqi, shogi, or chaturanga. Think it’s an edge case example? It should be assumed “normal”, modern chess? Well, do you know all Laws of Chess?

Requirements cannot cover everything

If you clicked the link above,  this is what you’d see first on the page.

The Laws of Chess cannot cover all possible situations that may arise during a game, nor can they regulate all administrative questions. Where cases are not precisely regulated by an Article of the Laws, it should be possible to reach a correct decision by studying analogous situations which are discussed in the Laws. The Laws assume that arbiters have the necessary competence, sound judgment and absolute objectivity. Too detailed a rule might deprive the arbiter of his freedom of judgment and thus prevent him from finding the solution to a problem dictated by fairness, logic and special factors.

Too detailed rules might cause damage, having a freedom of judgment is critical

Continuing with citation above: not only requirements and rules cannot regulate everything, if they try to do so, they begin an obstacle in finding solutions and moving towards goals.

Instead, context-specific, experience-based solutions – heuristics – should be thoughtfully applied for finding a solution.

Know who you are playing with

Every player has his own style. Knowing it helps to win. Studying it helps to know :)

Know your stakeholders! Their interests, their style, their goals. Knowing that you can shape your own tasks and actions in order to reach your goals and the big common goal.

Know how to estimate and budget your time

There is a whole set of rules on Chess Clock. The core idea: you cannot think unlimited time; you must act.

We’re on the clock in testing as well. Thinking too long ends up in being too late. Additionally, we never have a complete awareness of the situation.

Game Tactics

Is there a problem?

Whenever a move on the chessboard happens, players assess the new information. Where it leads? What is the threat?

Whenever testers discover a new piece of information about the product, “is there are problem?”, – they ask themselves.

There are no single best strategy, but there are good tactical moves

These moves come from the experience. Tarrasch rule, Lucena position, Sicilian Defence… These moves are usually right thing to do, but not always. There are exceptions. There could be better moves.

What this reminds of?  Heuristics, again.

Recording moves

1. e4 e5 2. Qh5?! Nc6 3. Bc4 Nf6?? 4. Qxf7# 1–0

Recording moves is common practice in chess. It has very well developed notation.

Sadly, in testing it’s often replaced by following a pre-recorded testing steps. But even such conservative tools like HP QC now offer additional capabilities for run-time note taking.

Game Analysis

“In chess, analyzing your games is a main task in your training toward mastery.” – This is a quote from article “A hardcore guide to analyze your chess games in 12 steps“. I’ve just found it by googling “chess game analysis”. And you know what? It’s so amazing, it just all applies to testing with minimum words replaced.

Why should we analyze our games?

Because…

A- We can compare our decisions against concrete evidence.

B- We can look deep inside our thought processes.

C- If done correctly, we will learn about the openings we play, the structures presented in those openings, and the endgames that we never had a chance to practice. If the analysis is completed new ideas will appear and you will know a little more about yourself when started.

D- When you analyze your games properly, all your mistakes come to the surface (knowledge, mental processes, recurring errors, etc.), and you will be one step further towards correcting them. The sweat you spend analyzing is worth its weight in gold.

The guidelines for analyzing your chess games

1. Write down the time spent in the game on every move as you play.
2. During the game create code signs on your score sheet to point out the critical moments of the game.
3. Immediately after the game, write down what you remember from the game.
4. Write at least three things you learned from the game you just completed
5. (IMPORTANT!) Identify the critical moments of the game …
6. After establishing the critical points, you can begin to analyze them.
7. Check opening theory
8. Positional patterns searching.
9. Analyze the game by yourself (and know when to use programs…)
10. Reports and diagnostics.
11. Publish your analysis.
12. Time to wrap up and verbalize the acquired knowledge.

Each rule has explaining paragraph in the article. It’s worth reading, believe me.

A closing note

With all “testing vs. checking” debates occurring and reoccurring, one might ask how far my analogy goes. After all, computers can play chess, and can do it pretty well. So, isn’t it a proof that testing can be completely automated, and tester-machine will beat even a master human tester?

My answer here is to consider a few major factors.

Artificially limited vs. unlimited and evolving context.

Even though quite complex, Chess is a very formalized and 100% predictable game. There is huge, but finite set of combinations of moves, which can be won through a computing power, similarly to brute force password hacking.

Even tiniest software development project will be undoubtedly more complex context, with ever changing rules, which also can and will be broken from time to time.

There is no way the machines can win by evolving in this direction.

Machines are unable to learn, so far.

In the famous “Deep Blue versus Garry Kasparov” match the whole team of engineers was working in between of the games, adjusting and improving its algorithms. Moreover, the grandmaster himself helped them to tune up the machine.

It’s not really the machine’s victory, but rather a triumph of engineering thought.

However, there is limited progress in this direction. Chess programs now use heuristic algorithms, dynamic algorithms, try to “remember and guess” opponent’s moves.

Theoretically, for a quite simple testing context such a tester-machine can be created, and, supported by a team of highly paid software engineers (programmers and testers) will be able to perform its testing duties at acceptable level. Wait! Wouldn’t be faster and cheaper ask those pro testers to provide test results? :)

The SkyNet factor.

Given that human may create machine-based artificial intelligence one day, it’s quite possible that it won’t be interested playing chess, but would want to win over the entire humanity.

This month’s top – Perfect Software and Other Illusions About Testing. Chapters 5 and 8.

***

Even though testers pretty much can/should be “jack of all trades” kinds of specialists, certain jobs they shouldn’t do as main and regular responsibilities.

• Secretary and administration – moving around papers and files, cleaning messes in repositories, shipping-receiving, taking-passing messages and faxes-mails, cleaning work areas.
  These are the kinds of things that everybody in the development team should be able to do properly, without leaving a mess. Or it must be a dedicated role, like office assistant, or person on duty according to the schedule involving all team members.
• Build and release management. Simply because it’s a critically important thing, and must not be done in casual ways. If QA department owns build and release management, it doesn’t mean that every tester can or should do it on his or her own way. This must be a regulated and controlled process.
• Support jobs. If any tester can be distracted any time for unknown amount of time, don’t expect them to be able to focus and accomplish testing missions.
  On the other hand, testers bring invaluable investigation and problem-solving skills to support team, and, in turn, learn more about real problems that customers are struggling with. Best working approach, as I’ve seen, is having a dedicated tester(s), assigned to help support team according to the schedule.

***

As the bottom line, all automation tool vendors, both commercial and open-source, promise the same or very similar benefits. What matters is will they be delivered, and to what extent. It most of all depends not on the tool, but the actual implementation. Once Michael Larsen and I put together an article on evaluation criteria.

After doing a number of reviews this year, I’d suggest “M” (which stands for Maintainability) to be made the biggest letter with 3 exclamation marks. Every time I hear of fascination with creation of scripts without a concern of how they will be used, I feel very sad about test automation.

***

Consistency with the purpose: Fail.

Since usability is one of the aspects we’re testing, I decided to mix in Personas in our testing approach – and started with defining a few basic personas.

The Impatient
Clicks quickly through the app. Sometimes confuses single and double-click, or repeats clicking the same command (link, button) if the response is not instant.
Makes mistakes, goes back, or cancels and restarts.
Makes quick judgments that often wrong.
Sometimes tries “new ways” or experiments with the functionalities.

The Cautious
Moves slowly and carefully through the app.
Always goes exactly by written instruction.
Expects everything to be precisely documented. Expects the application to communicate and instruct her during the work. Expects application to verify and validate everything, and the same herself as well, in order to avoid any mistakes.
Gets lost in unexpected situations; follows the pattern in confusing situations.

The Long Time User
Moves quickly and confidently through the app.
Knows all the limitations and has effective workarounds for them, but has some misconceptions as well.
Doesn’t need much of communication or confirmations as the sequence of actions is written in her mind.
Uses “restart and repeat” as a main solution for the all encountered problems.

***

Even though the original intent was to apply persona-specific testing style to evaluate usability, additional testing values have emerged.

Communication – testing as The Cautious helped to find spots where the application is doing poor job communicating with the user – either provides no confirmation, or brings misleading prompt, or even where the functionality is in disconnect with documentation.

State and Recovery – testing as The Impatient helped to find a few spots where the application does not restore into the same starting state upon cancellation, and continuing in that state would cause the problems.

***

I’ve got another hint from a Business Analyst, which might be related to Personas aspect of testing. Long time users are so used to the ways the app working, that even obvious improvements in the functionality might be seen negative, at least for a period of time.

Now, is that the kind of problems that testing should care about as well, and how can we anticipate and report (advocate) such problems?

While doing a lot of thinking, and having a lot of ideas I could write about, and having experiences I’d like to share, I can’t help but notice that I’m not blogging for quite a few months. It seems that I simply don’t have enough time to sit in WordPress, and even when I try to, an urgent matter comes up. And yet I had a lot of stuff written in my tester’s notebook. I do it in between of things. I take notes for the very same reasons as in the article linked. In short, note-taking is my thinking tool.

So I decided – as much I’d like to write elaborated ideas and experience reports, best I can do now is to publish some of my notes, and (hopefully) I will expand them into articles in the future. Or not.

***

Who should have a control over test environment? Programmers or testers? Seemingly, agile development teams shouldn’t even have such question. And yet I saw it existing in variations both in agile and waterfall teams.

There is a number of reasons why testers need the environment to be in a specific way.

• They want to finish testing mission while pushing new code will render all testing progress irrelevant.
• They want to know what code and configuration changes are being pushed in order to define their testing mission(s).
• The database in testing environment is too trivial, not helping to create situations that will/might happen in production.
• They want to have open access to all configuration and data in order to create testing situations by purpose, not by chance.

Not all of these problems demand for the ownership of the test environment by testing team. Some of them can be resolved by timely and precise communication. Some of them can be resolved by effective collaboration.

***

“The product is a solution. If the problem isn’t solved, the product doesn’t work. “

Hence one of the questions I’m always trying to clear – what problem(s) this particular functionality is trying to solve.

Problem – Solution – Logic – Implementation.

Logic is “requirements”. They can be very elaborate and signed off, or go quite informal as “user stories”.

Implementation is specification and the software product itself.

The problem is often in “Problem – Solution” pair. If a tester doesn’t realize what is the problem that the functionality intends to solve, how he (she) can go above and beyond functional correctness?

The most important and the most hard-to-find-and-expensive-to-fix bugs are not in the compliance with specification. They are gaps in logic, conflicting requirements, misunderstood (misinterpreted) logical rules, and, of course, inadequate solutions.

On the spectrum from Problem to Implementation, some roles seem to be involved only in certain phases – “requirements gathering”, “architecture design”… Testing, especially if it pretends for the Quality Assurance role, has to spread itself everywhere.

Description

1. Get parent object reference. In the example given, it’s a browser process, but you can narrow it down to a page or a web container object (table,  div, etc.).
2. Use FindAllChildren to retrieve array of links.
   - You can limit search depth according to your context.
   - You may want to do filtered search using additional properties: Visible, VisibleOnScreen, ContentText, etc.

Implementation

  Dim PropNames, PropValues, IEProcess
  Dim Links

  PropNames = Array("processname", "index")
  PropValues = Array("iexplore", 1)

  Set IEProcess = Sys.FindChild(PropNames, PropValues, 1, True)

  PropNames = Array("ObjectType")
  PropValues = Array("Link")
  Links = IEProcess.FindAllChildren(PropNames, PropValues, 100, True)

  PropNames = Array("ObjectType", "Visible")
  PropValues = Array("Link", True)
  Links = IEProcess.FindAllChildren(PropNames, PropValues, 100, True)

When TestComplete 9 was released, many (including me), complained about dramatic performance degradation in real-time GUI recognition mechanism (using FindChild family of methods). Though some parts of recognition chain started working actually faster, the first step – acquiring process handle – was taking very long time. Remarkably, SmartBear’s Support Team responded very quickly and initiated its own investigation based on the code examples provided. They even quickly released a patch. However, the solution is a complex treatment. Automation engineers need to know of the mechanics of the new GUI recognition engine and settings that tweak it.

How It Works*

* Parts of it I learnt myself through years of working with TestComplete, but I’d like to give a main credit to TestComplete support engineers who shared insights of the inner mechanics.

Recognition through properties

The tool does not actually “see” picture on the screen. It uses specified (pre-captured hard-coded or dynamically calculated) expected values of the properties of GUI-representing objects to find a match. Once an object with matching properties is found, the tool gets a handle of it, but still does not “see” any visual representation.

Parent-Child hierarchy

GUI representation follows a “parent – child” hierarchy. Window contains a button, button contains a label. A parent can have many children, but each child has only one parent.

GUI recognition properties have to be a unique set in the context of the parent if you want recognition to be reliable.

Depth-First Search

To find objects with matching properties TestComplete uses depth-first search algorithm which ensures quick exhaustive search down the chain of parents but makes the search longer if the target object belongs to parent chains further in the hierarchy.

Handling objects

Although software applications are unique on their own, presentation layer components (windows, buttons, lists, text boxes) are essentially the same and come in a finite small number. Yet the challenge of an automation tool is to recognize a particular class implementing the GUI element in order to get a hold of its properties and methods.

On the abstract, wrapping class for all the buttons is a single object, but in the underlying the wrapper must support button classes implemented in different libraries, for example, Dot Net and Delphi. For that purpose, TestComplete has “extensions”, loading of which is controlled.

What You Can Do*

* Use combination of these methods to find a setup best meeting your needs.

Disable unused extensions

Go to “File \ Install Extensions” and revisit the list – its pretty big. If you for sure don’t work with certain application types then disable loading of extensions for them.

Limit search depth

Whenever applicable, limit the depth of search. To see the difference in performance, you can try the following example:

  Dim PropNames, PropValues, IEProcess

  PropNames = Array("processname", "index")
  PropValues = Array("iexplore", 1)

  Set IEProcess = Sys.FindChild(PropNames, PropValues, 100, True)

  PropNames = Array("processname", "index")
  PropValues = Array("iexplore", 1)

  Set IEProcess = Sys.FindChild(PropNames, PropValues, 1, True)

Use Process Filter

You can instruct TestComplete to “ignore” all but specified processes. Go to “Tools\ Current Project Properties” -> “Open Applications\ Process Filter”.

The effect of this setting is that TestComplete doesn’t have to scan GUI of other processes and doesn’t have to load wrappers for all kinds of objects.

This setting might give you the best performance win; however, it can only be switched manually, which is inconvenient while running a batch that tests application in different browsers.

Description

Let’s say we need to retrieve a number of megabytes from under “Lots of space” category.

First, we need to explore the GUI implementation. If we’re lucky, this text node may have a unique class or ID value. Then it becomes a one-step search as in the code example below.

However, if the text node doesn’t have distinct identification properties we have to locate it through one of parents/grandparents that can be identified reliably. Then in the context of the parent we can locate the target text and extract it.

Implementation

  Dim PropNames, PropValues, IEProcess
  Dim objSpan1, objParent

  'Before running this code example make sure Google Mail Login page is up in IE

  PropNames = Array("processname", "index")
  PropValues = Array("iexplore", 1)
  Set IEProcess = Sys.FindChild(PropNames, PropValues, 1, True)
  If Not IEProcess.Exists Then
    'not found - exit
  End If

  '1 - Locating span by html ID
  PropNames = Array("ObjectType", "idStr")
  PropValues = Array("TextNode", "quota")
  Set objSpan1 = IEProcess.FindChild(PropNames, PropValues, 100, True)
  MsgBox objSpan1.contentText

  '2 - Locating span via parent object
  'Get parent
  PropNames = Array("ObjectType", "tagName", "className")
  PropValues = Array("Panel", "div", "product-info mail")
  Set objParent = IEProcess.FindChild(PropNames, PropValues, 100, True)
  'Get paragraph in parent
  PropNames = Array("tagName", "contentText")
  PropValues = Array("P", "*megabytes*")
  Set objSpan1 = objParent.FindChild(PropNames, PropValues, 100, True)
  'Little parsing to retrieve the number
  MsgBox Mid(Left(objSpan1.contentText, InStr(objSpan1.contentText, "megabytes")-1), 5)

Description

“Mouse Over” event triggers responses like highlighting, pop-up hints, etc.

• With TestComplete we can use one of GUI object’s methods – HoverMouse – to simulate mouse over by placing mouse cursor over object.
• Alternatively, we can use LLPlayer object to simulate mouse moves.

In the example provided both methods are simulating mouse overs on Google Search page.

Implementation

  Dim PropNames, PropValues, IEProcess
  Dim objSearch1, objSearch2
  Dim Iter

  'Before running this code example make sure Google Search page is up in IE

  PropNames = Array("processname", "index")
  PropValues = Array("iexplore", 1)

  Set IEProcess = Sys.FindChild(PropNames, PropValues, 1, True)
  If Not IEProcess.Exists Then
    'not found - exit
  End If

  ' "Google Search" button
  PropNames = Array("ObjectType", "ObjectIdentifier")
  PropValues = Array("SubmitButton", "btnK")
  Set objSearch1 = IEProcess.FindChild(PropNames, PropValues, 100, True)
  If Not objSearch1.Exists Then
    'not found - exit
  End If

  ' "I'm Feeling Lucky" button
  PropNames = Array("ObjectType", "ObjectIdentifier")
  PropValues = Array("SubmitButton", "btnI")
  Set objSearch2 = IEProcess.FindChild(PropNames, PropValues, 100, True)
  If Not objSearch2.Exists Then
    'not found - exit
  End If

  '1st method
  For Iter = 1 To 2
    objSearch1.HoverMouse
    aqUtils.Delay 500

    objSearch2.HoverMouse
    aqUtils.Delay 500
  Next

  '2nd method
  For Iter = objSearch1.ScreenLeft To objSearch1.ScreenLeft+objSearch1.Width
    Call LLPlayer.MouseMove(Iter, objSearch1.ScreenTop+Int(objSearch1.Height/2), 10)
  Next

Here are a few free data generator tools available online.

1. Lorem Ipsum
   It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using ‘Content here, content here’, making it look like readable English.
   You can specify amount of data to generate in paragraphs, words, bytes.
2.  www.fakenamegenerator.com
   To generate detailed “personal records” – name, gender, date of birth, address, credit card, phone, email, occupation, and many other data fields.
3. Wikipedia
   Although it’s not a data generator by purpose, it’s multi-language articles come extremely handy.

Description

• Returns text value of node’s child attribute specified by name.
• Valid arguments: an XML element node (NODE_ELEMENT type), name of the attribute.
• If XML node is not defined, returns empty string.
• Uses private function p_GetXMLElementAttrText, defined within the library (see parent page).

Implementation

Public Function ChildAttributeValueByName(ByRef objXMLNode, ByVal sAttrName)
  If objXMLNode is Nothing Then
    ChildAttributeValueByName = ""
    Exit Function
  End If
  sAttrName = Trim(sAttrName)
  ChildAttributeValueByName = p_GetXMLElementAttrText(objXMLNode, sAttrName)
End Function