The job of software testing, we know, is to provide information about the quality of a product.
Many people, however, believe that the only way to get such information
is to execute the software on computers, or at least review code.
But such a belief is extremely limiting.
Gerald Weinberg, Testing Without Testing

We got a mixed feedback on the last Weekend Testing Americas session – as well as we had a mixed group: many first-timers and frequenters.

Some comments were made by inspired people.

That was an interesting discussion and another indication that testing starts as soon as you start thinking.

I enjoyed it immensely. It was a “get my brain thinking in a different way” good.

It shows you can do a lot of testing without getting hands-on

Some comments were a mix of appreciation and surprise.

I was expecting something with more practical hands-on testing and this was more like a brainstorming session

Yes, it met my expectations in that the group was given a mission and came up with testing-related questions to figure out a plan of attack so-to-speak, but I was also expecting hands-on.

And there were comments I’m not citating because they were made privately by disappointed participants who wanted “hands-on testing and bug reporting” type of session, where people “actually test” instead of “discussing”.

The latter made me feeling slightly upset and disappointed, too.
Michael Larsen (see his write-up) and I put a lot of time and effort preparing each session. We don’t do any “teaching” at WTAmericas, we provide a ground for collaborative learning. We see our responsibility, as facilitators, to frame and structure a session in such way that it prompts learning of an aspect of testing… Anyway, I wasn’t sure whether we did our best setting the mission/steering the session, so my write-up is dedicated to how we see the testing craft and what aspects of it we consider the most important.

What Is Testing?

“Testing is a process of exploration, discovery, investigation,
learning, and analysis with the goal of gathering information
about risks, vulnerabilities, and threats to the value of the product.”
Michael Bolton, DevelopSense

Testing is multi-dimensional. Testing is open-ended. Testing is an aggregation of skills with a dynamically changing importance of each one. Yet there are key elements in testing. Remove them – and testing is not happenning any longer. Here are my thoughts on that.

Observation and Thinking

Testing begins from a sapient observation [1]. That is – what, when, and where to focus the attention is being decided in real-time, through conscious and unconscious thinking. Based on the new details observed, a tester may re-focus and re-prioritize – and the whole process of a sapient testing can not be pre-scripted as we are dealing with the unknown.

The observation might be controlled by a tester to some extent or he/she may have to be in a passive observer role, having no means to interact with a subject-under-test [2]. Does such a limitation affect the testing abilities? Probably, yes. Does it limit the thinking abilities? Absolutely not. If anything, such a limitation may even boost your thinking abilities because you remain focused.

And this is the reason why WTAmericas session N9 was structured so. Michael Larsen and I wanted to provide a ground for a focused observation, thinking, collaborative brainstorming and model-building.

Models In Testing

Functional modeling is what I studied at the university, along with the other disciplines of General Systems Theory. In my work I apply what I studied – implementing a model-based automation approach.
However, I comprehended the full power of models in testing only with my study of a heuristic-based approach [3].

Models are very useful in learning and problem-solving [4]. But any model is always a speculation to some degree. Even if it’s based on facts – those are facts about a certain behaviour in a certain context. Through observation we may add more facts to a model, and this sometimes results in re-shaping and re-building of the model. Models are useful in finding gaps and inconsistencies. Models help to come up with new questions. All these is used in acquiring the domain knowledge.

How Much Of Domain Knowledge You Need To Do Testing?

In short, I would answer – minimal, down to zero, unless all of “testing” is merely a checking.

Why is that so? – Because testing is learning. You don’t have to know something about the subject to start learning about it – although you can always learn some more. And this is why knowledge that domain experts have is nothing more than models. The models might be more accurate, more in-depth, more organized. The models can be used as oracles to answer questions that testers ask.

Testing Is Questioning

There is a thought that the purpose of existence of testers on a project is verification and bug reporting. Bug reports are considered the main deliverable (although, on some projects it’s test cases). Bug reports are counted and used as “performance” metrics.

Context-Driven School of Testing has a broader view on what testers find and deliver [5]. And it’s okay to do testing without delivering bug reports. Test models and test questions might be more important deliverables in the context. And the skill of coming up with the important questions is one of the most critical in any testing context. So is our mission as testers to work on sharpening this skill.

Yes, That Was The Mission

The main mission of Weekend Testing Americas is development of testing skills. And there is a lot of opportunities for learning. Communication and networking. And even for building bug-hunter’s portfolio in “hands-on testing” sessions, as it was with the previous one.

However, having a tool or not having a tool, going hands-on or relying mostly on observation is the matter of context.

And so we will keep going with all kinds of testing sessions.

References

[1] Sapient Processes

[2] A Transpection Session: Inputs and Expected Results

[3A] Heuristic Test Strategy Model – Satisfice, Inc

[3B] Heuristic Risk-Based Testing

[4] Speeding Up Observation with Models

[5A]  What Testers Find

[5B] More of What Testers Find

[5C] More of What Testers Find, Part II