Efficiency means different things in different contexts
The challenge of the hidden picture puzzle
Each click is a test
How much testing do I do before I decide I know what the picture is
Explicit Parameters in Testing
Coverage
Time
Outcome
Implicit Parameters in Testing
Progress
Correctness
Risks
Efficiency and effectiveness mean different things, depending on what you want to achieve and what you care about along the way
SNACK
Structure
Entertaining introduction
Main content
Entertaining conclusion
Invitation to CAST 2011
Narrative
Summary
James Bach and Jonathan Bach “escape massive nuclear explosion” and catch an argument on meaning of “efficiency”
Description of a testing educational game – hidden picture puzzle
Examples of different gaming styles with analysis and commentary
Making point about meaning in the context
Invitation to the conference
Highly professional narrative in the main content
Addressing
Audience: testers and other IT folks who like to do the job thoughtfully
Addresses in a thought-provoking manner
Aims at a highly relevant problem
Consistency
The video is highly aligned with the title and the purpose
Elements of “acting” in the entertainment parts
Professional, well-structured, well-paced video and speech
Keywords
The video operates highly relevant testing and test management keywords
No jargon, the video could be easily understood by a general English speaking audience
Comment – Argue – Acknowledge
Hidden picture puzzle is a reversed analogy of testing
Product picture – explicit and tacit models – is generally known; with puzzle it is unknown
Product testing uncovers problems in seemingly known picture, bugs are subjective, decision rules are heuristics; puzzle exercise uncovers picture itself, not bugs, and decision rule is literally black-white.
Test is a discovery of information
Depending of the actual test, it takes different time, skills, and means
It also takes a skill to decide what tests to perform
Value of information discovered may vary from critically important to useless
It also might be too late to know
Tests that confirm known information add very little value
I’m not sure I agree with the statement that Michael Bolton took a lot of risk
The goal was achieved – picture was uncovered and read correctly
Uncovering any more pixels (i.e. doing any more tests) in the given situation wouldn’t have added new information
Risks are not equal. Saying “took a lot of risks” when what are the risks is not defined makes little sense
Not testing product on officially unsupported browser version – is it “taking risks” or assessing risks and making conscious and justified decision not to test?
The graph presented in the video, frames “High Coverage – Little Time” testing style very attractively
I think, it’s a problem, because it misrepresents testing – it misses costs, and equates slapdash clicking with testing
How can “High Coverage – Little Time” be achieved?
Have more testers discovering information in parallel
More staff – higher costs
Have programmers dedicating part of their time to testing
Less programming done + switching costs
Increase efficiency of testing, remove obstacles and reduce bureaucracy
Easier said than done. Such a change can’t happen overnight
Downgrade testing to checking and automate it
Value of information is downgraded
More risks
Has investment and maintenance cost demands
Acknowledgement – even though I put a number of arguments from my side:
It was a really fresh and unexpected idea to model testing in such way, and to turn it into hands-on / minds-on educational exercise!
My main takeaway is about being explicit when considering these factors in my test strategy, making sure to discuss them with the stakeholders
The video itself is a long standing asset. I’ve used it for training sessions with my team and as a ground for time-coverage discussion with management.
Actually, that’s how I discovered the problem I mentioned above.
