Test Log or Test Results?

Well, I’ve got to start from definitions.
Traditionally, Test Results is a table, enlisting Test Cases along with PASS/FAIL status. While one might think that PASS/FAIL status is the only important information, for Testers is more important to know what have been covered.

Unfortunately, Test Case has no such granularity in details to provide information what was observed, what was considered, what was assessed, and what was verified. And, probably, it is impossible to provide such information as same tester never conducts same test case exactly the same way.

But testing script does. It’s not an advantage at all. And with scripts, reporting PASS/FAIL status only, it’s a HUGE disadvantage. Because a person, reviewing automated test execution results has no idea where scripts have been and what they did.

Not really agree? Let’s take an example.

Imagine, you want to take an outdoor trip. You ask your friends, where they went recently.

1. “I was at Rocky Place. It was OK. I have absolutely nothing to tell you besides that”.

2. “I was on Shadow Lake . Water was cold but it was OK. I have absolutely nothing to tell you besides that”.

3. “I went to 6 hour fly fishing tour. After 2 hours of driving North-West we parked at Entry Place (parking fee – $10 a day). Those, who needed, did a little shopping in the small convenience store there (no kosher food). Then we split into groups of 2-3 people, crossed camping area, and entered willow jungles. […] My second brookie was a tough fighter. After series of jumps and run-arounds I landed her, and carefully weighted. 3 1/2 Lb! […] That was a remarkable trip.”

Now, what can you possibly say about first two places? Well, you heard something about cold water, right? But do you have an idea whether it was cold for your friend, or colder than usual? Was it cold for swimming or fishing? How long did those trips take? Can you afford them? How should you be prepared?

I guess, I asked enough questions for you to come back to conclusion about test automation:

PASS/FAIL status (aka Test Results) is meaningless information.

In the next blog post we will get back to fishing story example to review what kind of information, and in what format, makes the story interesting and useful.