Notes on Testing Challenge debrief
I’m very glad that the testing challenge I set went to a second round.
I prompt my readers now to read through the post in Darren McMillan’s blog.
Please pay special attention to Results and Debriefing sections. I think those are great examples of facilitation to learn from.
I also would like to share what I consider important lessons from debriefing.
There is one important trick I didn’t mention in my blog when announcing testing challenge: distractions. As in our work we balance with multiple goals and have to workaround various obstacles, while testing the game contestants had to deal with it too.
With this challenge the most common distractions, as I picked up from the debrief, were:
- heavy documentation approach while being aware of a shortage of time;
- focus on areas without critical impact (soak and performance tests) (considering short game time and absence of any important data, even crash of the browser is a low severity).
Another two points for consideration would be:
- focus on “traditional” testing (i.e. functional, usability) while missing Internet-specific critical areas, Security first of all. (What if this game is doing something else?);
- framing effect: testing through direct interaction with the program only. One step out would be playing with the size of a browser window, further away – exploring indirect and non-user inputs (like changing local time).
Overall, I think it went great, and let’s thank Darren for sharing this fun with us!
One response to "Notes on Testing Challenge debrief"
Albert, thanks for this follow up post :-)
Soak testing I never thought of this way, your right the game time will be short. Lets consider however if we left our browser on the page for a few hours or overnight & came back to find a memory leak had crashed the browser? I do agree though we have to think about critical impact, excellent observation, the soak test was a freebie from an flaw in the game so in this case I think it was acceptable to do this. Like you say though if we’d spent a lot of time focusing on this with only two hours to test we’d have been falling into a testing trap, as the majority will play for a short period and move on.
Security was listed in the recommendations, but not tested in the time frame, it could have been but I think this is more from the lack of skills in our team for security testing. It’s something we as a team need to improve upon.
Your local time catch was excellent :-) You could have found this just by messing about & trying things, but no you followed the code & exploited a flaw you discovered to your advantage. That’s brilliant :-)
Thank you for coming up with a brilliant challenge :-)
[ Albert’s Reply. Thanks, Darren. In your blog you mentioned you’re gonna set up another challenge. Please sign me in! ]