Heuristics exposed
Software Testing is an empirical process.
Empirical approach as an opposite to theoretical requires observation, investigation and experimenting.
Results of those actions performed are collected, documented, analyzed, and categorized, before conclusion is made.
A Scientific Method of Learning consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses. If hypothesis is consistently confirmed in series of reproducible tests then it is considered truth. All the data, ideas, and methodology should be documented and shared so they could be objectively reviewed by side observers in order to verify results by attempting to reproduce them.
Everyone familiar with Software Quality Assurance Testing and its role in Software Development Life Cycle will see a close match between it and Scientific Method.
There is, however, one major difference. Scientists research physical laws, that never change and apply everywhere in the Universe. For testers, every new program build is a new “universe”. We have to research its qualities all over again.
Researching of computer program involves two kinds of processes:
-
process of verification of known functionalities to see if they meet defined business and technical requirements;
-
process of investigation of unknown functionalities and side effects in order to reveal their qualities, influence to and interference with defined functionalities.
The problem here is that known functionalities are almost never defined at the finest granularity level. There are always gaps and generalizations that filled with developers’ assumptions or even left missing. And even if requirements are defined at the best possible precision level there is always possibility of miscommunication or misunderstanding. As if that’s not enough, even in ideal situation if it’s coded exactly as specified there is a good chance that it will be messed up by the next release. Ironically, the current release is also still at risk because computer programs are sensitive to hardware and OS configuration, and dependant on their settings. So if a test is passed in the test environment it is not 100% guaranteed that the tested functionality will work in production.
Investigation based methods are in better place as they do not rely on expected results but objectively provide information on program’s behavior. Furthermore, decades of worldwide practice demonstrated that detection-oriented approach results in better software product’s quality rather that verification-oriented.
While staying “Black Box” methods testing heuristics consistently allow finding defects in program code.
In the subsequent posts under this category I’m tracing the roots in order to find out what are that weak spots and how they appeared.
// posts are under construction, links below are temporarily unavailable
- Number Boundary Testing
- Data container boundaries
- Complex data boundaries
Documentation: Publications and References
“Testing Computer Software”, Cem Kaner, Jack Falk, Hung Q. Nguyen
“Lessons Learned in Software Testing”, Cem Kaner, James Bach, Bret Pettichord
Heuristic Risk-Based Testing by James Bach http://www.satisfice.com/articles/hrbt.pdf
Test Heuristics Cheat Sheet by Elisabeth Hendrickson http://testobsessed.com/2007/02/19/test-heuristics-cheat-sheet/
One response to "Heuristics exposed"
All techniques in testing are heuristic. That is to say, there are no techniques that can guarantee you will find all important bugs in a reasonable period of time. There is no ALGORITHM that solves the problem of testing.
One example of a heuristic is cruise control on a car. It’s not a substitute for a driver. It just helps you drive. So to use heuristics properly (in other words, to use any test technique properly) we must understand them from the inside out. We must be able to pilot them.
I like your idea of looking to the source of every heuristic. That’s what you need to do.