“A buffet is a system of serving meals in which food is placed in a public area where the diners generally serve themselves.” – that’s from Wikipedia. The place where I’ve had lunch today was just like that, except the actual meal wasn’t the point. Instead, we served each other ideas and challenges on testing and diligently worked on them.

Now, I won’t be putting much of details here. I think, the best would be to come and engage in person. But for those who don’t live in Greater Toronto Area, here’s someting…

I really enjoyed conversation with Fiona Charles about the subject:

I had even longer discussion with Adam White, the organizer of the event,  and Michael Bolton, about the following proposition -

- but we still left it only partially covered.

I have made two major investments into my professional development this year.

One of them, taken in April, is the course of Rapid Software Testing. I wrote about my experience and takeaways, as well as proudly gave a personal endorsement.

Another course, similar and different at the same time, is Black Box Software Testing (Foundations level), which was taken in Autumn. Same as with the RST course, I’ve got a unique blend of experiences.  And I want to say more about it.

The course itself is organized in a “research lab” style. It actually reminded me of my happy student years. Theory and practice are given in an excellent mixture, and the course has a potential of keeping knowledge-thirsty students busy for 40 hours a week. (Although you don’t have to spend that much time to pass the exam. But I found the assignments so addictive that I worked even on those not given to me.)

In this sense, the Foundations level course will be a true testament of skills even for someone considering himself (or herself) a senior tester.

The course aims for the following learning objectives:

• Understanding key testing challenges; choosing the strategy to meet your actual needs under present constraints.
• Rapid learning, as well as learning through collaboration and reflection.
• Critical thinking and precision in reading.
• Efficient and effective communication, and well-structured feedback.
• Understanding diversity of definitions in software testing terminology.
• “Complete” vs. “Good Enough” testing.
• Heuristic approach.
• Programming fundamentals.
• What is coverage?
• Measurement and metrics. 

That’s a lot for just 3 weeks! But what makes the course really special is the tight collaboration it requires and encourages. I felt pretty strong on many topics yet I was deeply engaged in the assignments either finding new aspects for myself or supporting fellow students. We had quite a group! I wish at my day jobs I’d always meet such a strong passion for testing and learning.

If, at this point, you haven’t signed up for the course yet, here’s another argument to consider: you will get to work with a star team of instructors. For the particular course I was taking, our instructors and facilitators were Cem Kaner, Rebecca Fiedler, Michael Larsen, Selena Delesie, and Brian Osman. – And I’m very thankful to them for all the feedback and coaching given.

On a final note, I want to say: if the RST course with Michael Bolton was the most insightful and inspiring experience of this year, the BBST course was the most challenging and motivating experience, and I look forward to taking Advanced classes.

Publication URL: http://www.thetestingplanet.com/2011/11/november-2011-issue-6/

Download PDF: The Testing Planet November Issue

And you can view the large mind map here (click to open in a separate window).

Note. The reason of this posting is two-fold:

• While considering taking the Rapid Software Testing course I found surprisingly little of feedback information on testers’ blogs, and none of that addressed the questions I actually had. So I’m fixing this issue retrospectively.
• There are too many aggressive pseudo-education “courses” or scamming certification schemas advertised on every corner. I think it’s time for all the testers who care about the craft to start caring about signal/noise ratio. I do it with my personal endorsement.

And here it goes in a question-answer format.

- Why should I take the course if there are a lot of materials available online, for free? 

Yes, there are many articles, blog posts and presentations available online on James Bach’s and Michael Bolton’s web-sites. They are provided in a form of condensed knowledge. They are conclusions. And the course is rich with hands-on, practical exercises, which allowed me to digest the theory and acquire the conclusions as if they were my own. Plus the special exercises giving a chance to try what have just been learnt.

Yet the materials available online served another important purpose for me. Reading them, studying them, prepares for taking of the course at much deeper level, as there are many layers of knowledge there. 

-  What I wish was different?

 The first answer would be the length. I wish it was 5 days long. Some areas were touched too briefly. (Though Michael Bolton explicitly said that he leaves a lot for personal learning, and that’s what I’m doing quite actively). Anyway, if there was a “Follow up RST course” I would have gladly gone for it. Since we don’t have one, I channel my energy into personal learning and online collaboration with likeminded testers, especially at Weekend Testing Americas.

In technical aspects, I wish we talked more about multi-tier, large scale applications. That’s what I deal with at work, and that’s where I see – applying the heuristic-based approach gives especially critical advantages. 

- What this course does not teach?  (and some about what it does)

It does not teach abstract definitions and bookish techniques. However, it teaches to critically assess claims and statements without worshipping to labels and metrics.  This prepares to be comfortable and efficient as a tester in a variety of environments with their unique contexts and challenges.

The course teaches to develop and thoughtfully apply testing techniques as needed and as long as they make sense (dealing with the problem of “best practices”).

The course doesn’t teach testing politics or staff management. However, we were given some effective communication techniques that one can apply in all levels of the organization.

The course doesn’t teach either programming or ‘test automation’. However, Michael Bolton demonstrated some impressive examples on how use of tools and scripts amplifies effectiveness of tests. I’ve also learnt about the ways how an automation developer can increase his value for the testing team. 

- Which part I didn’t get? 

Or, at least, I’m not sure I got it the way it was intended.

This is test estimation part. I know from my own experience, and I totally agree with the concept that nobody really cares about “test estimation” – testing just absolutely must fit the development schedule (“Release date is written in stone!”). That is why, as a contractor, and a test lead, I pretty much do “test budgeting” trying to use the time given most effectively instead of demanding for the time budget based on the estimates. And budgeting time boxes for session-based exploratory testing – I get that. But how to plan “test execution phase”? How to bake in time for investigation of unknown problems and re-testing of unknown number of bug fixes? Possibly, these questions are addressed in “Rapid Software Testing for Test Managers” course? I’m definitely going to take this one. 

- What I found especially challenging in the course? 

The course is called Rapid Software Testing for a reason. Yet, I’m rather analytical type of person. You know – gather [all relevant] information, build a picture (model), carefully review, etc.. Of course, in testing, redundancy of information is a luxury one rarely has. As the course teaches rapid and fallible methods of assessment (heuristics) in conditions of lack of information and time pressure, I felt that my ‘natural habits’ anchor me. This was a bit frustrating at times. 

- Did I feel disappointed in some of my expectations about the course?

No, not at all. In fact, I gained more than I expected. However, I used to see some comments online expressing vague disappointment. With regards to those, I can say – the course is not a magical initiation which instantly turns students into testing experts. Instead, it shows the lead and charges with the inspiration to follow the road of continuous learning. And that’s the most valuable takeaway. 

- What I have gained I didn’t expect at all?

I have read some wide-known books about testing and even much more online article. I heard of the notion of a ‘testing story’ in online talks within the context-driven community. But the course gave me clear and concise points – what is that, why is that important, and even how to do that – telling testing story.

-  To whom would I recommend attending the course?

 Myself, 5 years ago. 

Another answer: anybody involved in software development, especially testers, programmers, business analysts – and their managers.

Modern applications have a few layers of ‘defense’ that are supposed to protect against unauthorized access. The very first layer is at the front-end, merged with the UI. The UI part, client-side part, is at user’s computer and therefore users have the most capacities to manipulate it. Penetration testing is a purely exploratory testing activity. By using trial-and-error approach we can learn about particular weaknesses in the application, and then use that knowledge to gain some advantage. Of course, as testers, we use that knowledge to evaluate the risks and to help improving the application’s security. Although complete security testing requires specific knowledge and skills, there’s no reason why Black-Box testers can not expand their arsenal with some quick, simple, yet powerful penetration testing techniques.

Groundspeed is a free tool that allows to learn about application and to manipulate it in the ways users normally can not, thus helping to expand the testing scope.

Recently, I conducted Weekend Testing session #21 where we worked through a few exercises, and then discussed new testing techniques enabled by the Groundspeed.

1. Bypass front-end validation of values.

2. Input values that you are normally not able to input – via hidden fields.

3. Trigger function calls in the application that are normally disabled or not accessible.

An unexpected outcome was a shorthand mnemonic crafted by Michael Larsen: “Remember the BIT”.

Archery is a craft, practice, and skill. I dedicated a fair amount of time in my youth when I practiced sports archery, and I still like Medieval age novels. Needless to say, now I dedicate a lot of my time to the craft of testing – but there are analogies to archery that we can use…

In the scripted approach to testing everything is dominated by numbers. You know, “10,000 test cases in scope”, “100 days to ‘execute’”, “20 test cases per day per resource”. Well, in Medieval Age some commanders tried to follow similar ‘straight forward’ approach: “1 knight is as strong as 10 regular soldiers”, “3 to 1 strength ratio guarantees victory”.
Yet as History knows, a straight forward approach almost never worked. There are too many factors that may throw off balance and screw up the most careful planning. The presence of Longbow Archers was one of them. At the Battle of Cressy, heavily outnumbered Anglo-Welsh army was victorious as a result of superior training and tactics of Longbow Archers.
In exploratory testing, we are constantly sharpening our skills, and we use asymmetric approach to overcome outnumbering hordes of bugs.
I have been meaning to offer fellow Weekend Testers some armour-piercing practice for a while, but I was held by inability to bring in some solid testing challenge. Finally, my thought is – that shouldn’t prevent us from conducting a group training session. I will offer examples I have prepared, and we can together discuss the techniques we want to include in our arsenal.

How to participate

1. Add “WeekendTestersAmericas” to your Skype contacts if you haven’t already.


2. Fifteen minutes prior to the start of the session, message “WeekendTestersAmericas” and ask to be added to the chat session. Once we see you, we will add you to the session.

Also recommended: Send an email to WTAmericas@gmail.com with the request to add you to mailing list to receive announcements.


Be prepared

Install Firefox Browser.

Install GroundSpeed add-on for Firefox.

Be on time

Date: Saturday, November 5, 2011

Time: 09:00 a.m. – 11:00 a.m. PDT. Check in your time zone.

Being a facilitator at Weekend Testing Americas I often invite people to participate in our sessions. I get all kinds of responses but one is particularly interesting: “I would participate in a session if you give me test cases to execute”.
I answer differently to this one. I may suggest to try designing and executing test cases at the same time. I may suggest to join a group learning session when we try a new tool or testing technique. What I do not say is something like: “no, if you want to participate, you must do exploratory testing.”
You know, why? – Because one needs no theory to start trying and discovering. Moreover, knowledge gained the hands-on way becomes an active part of experience, a natural reaction like in martial arts. The important part is to begin.

So, what’s this preamble for?

I was ‘flying solo’ as a facilitator of the last session.
The founder and restless first facilitator of Weekend Testing Americas Michael Larsen was about to fly away to the AST Board Meeting, so, when I suggested to hold another session he suggested that I do it alone this time. So I went and did.
I think, I did pretty well. It was a ‘no big deal’. Of course, there was a big ground work for that. Before that session, I facilitated other ones helping Michael, developing missions, conducting debriefings, and stepping in wherever and whenever needed.  In this sense, there was nothing really new for me. While keeping up with all the threads going is hard, because you can only multi-task this much, I already had a sense how much should I support one or another to keep it going. As I knew I’m going to be alone, I put more effort upfront designing the mission. And (I find this part especially great!) the remaining space was filled by leadership of participants.

I find it amazing how much we can gain by consciously learning from what we do. With such a skill anything is hardly a ‘big deal’.

In this post I’m sharing our up to date experience mindmapping team-based exploratory testing as well as stating some expectations that a mindmapping product should meet to fulfill our needs.

Structure

As you can see, GUI part of the structure is the most developed here. Mapping is pretty obvious.
However, we’ve identified the following problems.

• What if GUI is structured and displayed differently, based on the user roles, account permissions, etc?
• What if GUI is structured and displayed differently because the same user can customize it?
• What if GUI is structured and displayed differently, depending on the platform?
• What if GUI is structured and displayed differently, depending on the operations user might perform?

Mapping all the possible states on the same canvas makes no sense. And cross-linking all the possible dependencies on the same canvas makes the map too busy.

Function

Functions is another hierarchy that is fairly easy to map. However, the typical questions you might face are the following.

• What if two or more functions reuse the same subfunction?
• How do you link functions to operations?
• How do you link data to functions?

Time

The challenge for “Time” section is that any observations are highly context-sensitive. Saying like “it takes 3 seconds to execute regex” has no meaning. Even if you’d make it “the regex” with a hard-coded sample, there are still other dependencies, platform, to say the least.
We think, leaving this area as a set of questions (where one can link observations) is better choice.

Platform, Operation, and Data

Platform might affect everything, from Structure to Functions.
Operations involve using Functions.
Data can be categorized by type; samples can be given in the context of a function.

When looking at the overall map you’re probably wondering – where do value goes? where are the claims? where are the risks? Those are the questions we identified but still not sure how to map. The main challenge in mindmapping approach is in the simplicity of a mind map. The actual process and product have far more connections, the connections are dynamic, and variety of the information is beyond formalization. As so, we set the goal of keeping documented the reasonable and understandable part of knowledge – charters, session logs, bug reports, and summary reports. We believe that any participant who missed one or few sessions within the project will have a guiding structure to catch up, but will have to do exploration on his/her own to ‘load’ the process map into the mind.
We also think that as the project goes, all the mindmaps connected will form a meta-map, a 3D mindmap, storing the project knowledge.

Appendix – What mindmap application is needed to provide

Real-time mapping – viewable by the all participants right as it is being created.

Customizable Access Control. View only, View/Copy, Edit. Ideally, with back-ups and versioning.

Export. Image, PDF, and so.

Linking mindmaps to the mindmap nodes. Either in a “floating node” fashion or as a multi-tab document. Better – both ways.

Attachable content. Notes, text files, data tables and video files.

Custom markers.

Tags. With abilities to arrange by tag and search by tag.

In this post I share experience on how we’re building and using the process model; it’s an expansion of ideas from the previous post.

The picture above is based on the sketching we made with Michael Larsen while conferring on Skype. However, the elements and solutions used in there are rooted in Rapid Testing and BBST methodologies.

We have started with longer than regular sessions (4 hours instead of 2), and have done two of such kind.
While the initial exploration of the product goes very well (at pace much faster than I could map it :) ), further breakdown is slower and slower. This is where we came to the conclusion that charter design and charter execution better be separate activities if you don’t want the whole group working on a single charter. However, and a comment by Thomas Ponnet emphasizes that, it brings a challenge of having  numerous debriefing sessions, which can only be addressed by having more facilitators, which is a challenge itself. For example, I am, personally, only learning the skills of debriefing.

The next thing we encountered that, once ramped up into the brainstorming mode, the group works with an awesome productivity generating vast amount of test ideas. However, if you don’t want them to get lost, somebody has to take care of them, and also frame and shape test ideas into charters fitting the timeboxes.

The ‘charter’ thing. Pretty much, any and every test execution is a charter. Charters produce test ideas for new charters. Charters produce information about application’s behavior and product elements. Last but not least, charters result in bugs and issues found… But how to frame and categorize the charters? On the diagram above, you can see sample charters called like “explore”, “inspect”, “investigate”, “question”, and “examine”. I’ve been learning charter-based exploratory testing mainly through Michael Kelly’s materials, here’s one short and sweet example. Since we agreed to build our process around the product coverage model, charters directly related to that fit very well here. But not everything is about product elements. Here’s how we got the list of charter categories:

• Explore <coverage element, functionality, UI>. The most ‘generic’ type.
• Inspect <claim, statement>. Could be a documentation claim. But could be a charter to re-test a bug fix. Or clarify information from the previous execution of the same charter.
• Investigate <problem, bug, issue>. It’s a “drill-down” and “explore around” charter. Often is a follow up on the results brought by other charters.
• Question <functionality, outcome>. It’s a mix of exploration and investigation guided by the questions “did it solve the problem?”, “does it add value?”
• Examine <operation, data>. Exploration from the end-user perspective.

One might ask why there’s a need to categorize at all. At the end, all charters result in producing information of the all kinds. The answer is in the need to land the charters in a certain structure. If you let them hanging – you lose the big amount of them after a session ends. And since you store them – you better do it in organized fashion. In terms of mindmapping – how do you want to visualize a charter and where do you want to attach or link it?

One important thing I want to mention. Findings of all types – bugs, issues, product’s behavior, test ideas, etc. - are getting produced at any time and during any activity – and they are the most valuable process artifacts. Even if it’s not an actual charter execution there’s always time and place to log your findings.

Here I’m looking at the challenges logged in the previous post.

I wasn’t the first one saying that it might be good to have certain expansions of Weekend Testing format. What if we have longer sessions? What if we do follow-up sessions? I think these expansions help addressing challenges like “complexity of a mission” on an individual level.

However:

• any knowledge learned by a participant remains encapsulated – there is no contribution towards product coverage;
• the testing approach is not organized and is not systematic at the team level.

What about organizing into a team doing exploratory testing?
The immediate questions are:

• how to distribute and coordinate the testing effort?
• how to qualitatively communicate and store the knowledge?
• how to qualitatively retrieve and visualize it?

Michael Larsen and I had a few discussions on the subject, as well as we brought it up during session debriefings and discussed with the participants.

As for the first question, Session-Based Test Management methodology fits greatly here.

• Charters help to identify individual testing missions.
• Charter notes help to capture and communicate the knowledge.
• Debriefing allows plugging in the new pieces of knowledge into the picture (information about the product and the process) being built.

But here’s the problem. If the picture is in facilitator’s mind then effectively sharing and maintaining it in a distributed team is hardly possible. And the whole process depends on the one, initial, facilitator. We wanted to make the process more open and more self-sustainable. We didn’t want to “manage resources” or oblige the participants with some strict rules.

And here’s what we have about that.

• Test ideas appear as a result of the team brainstorming in the group chat
• Test ideas are to be shaped into charters with ‘short’ (30 minutes) and ‘standard’ (1 hour) time boxes
• Anybody can test out the charter but in order to have a debriefing conducted the results have to be written following a certain structure, so this knowledge can be shared

Important to mention that “test results” are not equal to “bug reports”. The latter is only a part of the information about the product; hence the notes must contain everything learnt about the product during execution of a charter.

But what structure and what medium should be used to store product characteristics?

- So far, we found that James Bach’s “SFDPOT” in a form of a mind map works very well.

• Mind map makes the information visual and structured
• Mind map can be fairly rapidly created and easily maintained
• Mind map can be expanded at any node and in any point of time
• Session logs, screenprints, and other files can be easily linked to the mind map nodes
• Mind maps can be cross-linked

With all the above said, we’re getting product testing oriented series of sessions of four types.

• Exploring and mapping product elements (format: common chat and mindmapping of the main elements in real-time)
• Charter design (format: common chat and attachment of charters to the mindmap)
• Individual and paired charter-based testing sessions. Note that individually some participants may work on charters even offline, not during the Weekend Testing session. This greatly increases flexibility and allow differentiated contribution.
• Charter debriefing. This has to be done online and must be immediately followed by “admin” work – updating the mind map, logging bugs, uploading charter logs.

To sum up. Taking on the bigger tasks requires transformation from “crowd mode” to “team mode”. Team means takeaway from facilitator-participant roles, and a bigger variety of roles. And to effectively share knowledge across a distributed team there must be a dedicated medium. We’re looking into putting in use mind maps as knowledge transfer and communication tools.