One of the exciting things about testing meetup is the mix of roles and experiences people bring in. This time we had a research workshop with a rare opportunity of getting feedback from Senior Program / Product Manager participating in our event.
My online readers probably have a clue about the theme.

Extreme Testing

Introduction

Am I adding value with my testing? 

What is the role of testing in the eyes of the Program Manager? What Product Manager expects from testing? What is really valuable? What doesn’t make a difference? Do test case counts matter?

We are going to run a research workshop using persona- and scenario-based testing tactics. We will have a rare opportunity to receive instant feedback from Sr. Program/Product Manager who will be participating in this session.

Our Special Guest:

Kathleen Chan, Senior Program Manager, Do Process

“No one knows the cost of a defective product – don’t tell me you do. You know the cost of replacing it, but not the cost of a dissatisfied customer.”

Our special guest Kathleen Chan will help us to learn about expectations that Business holds. She will also take part in the exercises and will be providing feedback on testing strategy and tactics being developed.

The Product

The Personas

Student

• On a budget
• Lives in campus or rents a room
• Has strict morning schedule on weekdays
• Goes to bed at random

Traveler

• Mostly business and consulting trips
• Travels across North America and abroad
• Travels take from days to weeks
• Getting up time may vary

Family

• Live in apartment
• Parents, teenager, and toddler
• Main user – parents
• Mother gets up 1 hour earlier than father
• Don’t need to get up early on weekends

Backpacking Tourist

• Takes 1-5 days long trips
• May stay in motels
• Takes hiking and cycling trips
• May camp overnight
• Need to be on time for bus and train departures

The Statement

“Here’s the working prototype of our new product. We need you to test it. You’ve got 3 days. What will be your testing strategy? Please be ready to explain it. Test cases and their numbers don’t help me.”

There was a remarkable range of reactions to the Program Manager’s statement about uselessness of test cases.

Till certain career level testers might simply be shielded from the questions like that. They’re just testing. Executing test cases. Business goals of the project, budgeting, and spending are completely beside their worries.

On the other hand, test case counts are often the main and only test management metrics. Test leads and managers are used to estimation in number of test cases, as well as tracking how many test cases were created and executed.

I could see what I’d call a Factory School pattern. They demanded complete a BRD (“Business  Requirements Document”) and time to review it before getting to any testing.

So there was certainly a bit of a backlash on different levels.

The Approach

The approach I suggested was to explore both the product and marketing personas. To discover and to tell the product story in the context. Beginning from it’s purpose of use.

We went through the whole P.E.R.S.O.N.A. presentation.

Then split up to teams so that we can explore different aspect of Purpose-Errors-Risks-Style-Operations-Needs-Attitude.

Of course, each of these dimensions are connected, so it really were “projections” like Purpose-Errors-Risks or Purpose-Needs-Operations.

For those unfamiliar with guided exploration approach there was a bit of confusion but they could follow along and see the results live.

The Reports

Reporting in a structured and concise way is a testing skill by itself. I feel there’s an idea for another workshop here. I could see that our Client at times struggled following the reports.

Reports also revealed few more patterns.

What I’d call Context-Driven Testing pattern was testing focused on the real product in the given context towards requested risks. Team members looped between the prototype and testing objectives. The Product Manager was brought in to clarify a few important concerns.

Agile pattern, as I’d call it, was broad narrow testing mixed with the design. Team members spent very little time exploring the prototype but produced many ideas how they see the product should be implemented. Actually, it seemed like every user group would get their own clock. They certainly had fun. And they kept Product Manager very involved.

As for the Factory School pattern, some group members kept asking why should they worry about the strategy if test manager/lead will give them directions. Or, better yet, request the business to provide the team with test strategy or at least ask the business to approve the strategy document.

Disclaimer

If your perception of testing schools or patterns is different from the examples above – feel free to comment and tell your story! If you happen to be in the Greater Toronto Area – we can meet, discuss, and run another workshop where you’d demonstrate the difference.

The Feedback

• Our Client certainly got interested in the approach where testing is constructed to explore and tell the story in the context of business purpose and risks. She noted that such testing approach provides the answers she needs.
• Our Client noted that with the guided structured exploration approach the testing activity seems to be scalable in a more sensible way than with test cases, and it’s important for big products and teams.
• Our Client politely let know that she didn’t request re-design and not sure if such time expenses are justified from business and schedule perspective. She noted that she’d appreciate more time spent testing the actual prototype rather than keeping her involved in the design of hypothetical products.
• Our Client politely declined the requests to provide and / or approve testing strategy. She noted that it would be strange if a professional asks someone else, non-specialist, for instructions to follow.