Software Testing is an empirical process.
Empirical approach as an opposite to theoretical requires
observation, investigation and experimenting.
Results of those actions performed are collected,
documented, analyzed, and categorized, before conclusion is made.

A Scientific Method of Learning consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses. If hypothesis is consistently confirmed in series of reproducible tests then it is considered truth. All the data, ideas, and methodology should be documented and shared so they could be objectively reviewed by side observers in order to verify results by attempting to reproduce them.

Everyone familiar with software testing and its role in Software Development will see a close match between it and Scientific Method.
There is, however, one major difference. Scientists research physical laws, that never change and apply everywhere in the Universe. For testers, every new program build is a new “universe”. We have to research its qualities all over again.

Investigating a computer program involves two kinds of processes:

• process of verification of known functionalities to see if they meet defined business and technical requirements;
• process of investigation of unknown functionalities and side effects in order to reveal their qualities, influence to and interference with defined functionalities.

The problem here is that known functionalities are almost never defined at the finest granularity level. There are always gaps and generalizations that filled with developers’ assumptions or even left missing. And even if requirements are defined at the best possible precision level there is always possibility of miscommunication or misunderstanding. As if that’s not enough, even in ideal situation if it’s coded exactly as specified there is a good chance that it will be messed up by the next release. Ironically, the current release is also still at risk because computer programs are sensitive to hardware and OS configuration, and dependant on their settings.  So if a test is passed in a test environment it is not 100% guarantee that the tested functionality will work in production. 

Investigation based methods are in better place as they do not rely on expected results but objectively provide information on program’s behavior. Furthermore, decades of worldwide practice demonstrated that investigation-oriented approach results in better software product’s quality rather that verification-oriented.  

Another good thing about empirical methods that you don’t have to spend (actually, waste) time and efforts creating Universal Laws. If something under some circumstances works more often than it doesn’t – you can use this approach. The approach may evolve as context and methods change; it can get abandoned and revived back. Any practical approach is a heuristic. With regards to software testing, I also want to link this definition to James Bach’s explanation.

Software testing heuristics are not bound to “Black Box” testing methods.

In the subsequent posts under this category I’m tracing the roots of different problems in software (bugs) in order to find out what are that weak spots and how they appeared.

Publications and References

“Testing Computer Software”, Cem Kaner, Jack Falk, Hung Q. Nguyen

“Lessons Learned in Software Testing”, Cem Kaner, James Bach, Bret Pettichord

Heuristic Risk-Based Testing by James Bach http://www.satisfice.com/articles/hrbt.pdf

Test Heuristics Cheat Sheet by Elisabeth Hendrickson http://testobsessed.com/2007/02/19/test-heuristics-cheat-sheet/