How to find bugs quickly? How to pick just right test scenarios without losing your time trying thousands of possible combinations?

These questions worth years of research.

In my post I’m going to cover just one simple approach I practice quite successfully, and give a few examples you might be able to reproduce.

Let’s start from definition.

Probability is a way of expressing knowledge or belief that an event will occur or has occurred. The concept has been given an exact mathematical meaning in probability theory, which is used extensively in such areas of study as mathematics, statistics, finance, gambling, science, and philosophy to draw conclusions about the likelihood of potential events and the underlying mechanics of complex systems.
Wikipedia.org

As you can see, almost everything from the definition above applies to software testing. Complex systems – that’s what we investigate. Mathematics and statistics – is what we use in analysis. Gambling? – Well, testing involves guessing, and the thrill of the bug hunt is for sure addictive. Overall, software testing is an alloy of science and philosophy.

Considering probability

Exhaustive investigation of all possible boundary conditions is very time-consuming.  In the same time, after any code change of the particular functionality all boundary investigation results become obsolete. That is, unless you have infinite time, the primary goal is not to find all boundary bugs but look until we find the first important one, and then move on to another piece of functionality (Why? Please read about “The Dead Horse” and other stopping heuristics).

And here is the rule of thumb I use.

Try first those Test Scenarios that

least likely were considered during implementation
and/or
much likely can happen during use of the system.

 Note, that these two probabilities are not the same number. It could be “not considered but much likely to happen” (worst case) or “surely considered and unlikely to happen” (happy case).

Examples

1. User Input Filtering and Validation

Let’s assume, there is a requirement: a user can enter only numbers in the text box.

You try keystrokes using every single button on the keyboard, and see that invalid characters are successfully filtered. Then you type an invalid combination in Notepad, Ctrl-C, Ctrl-V, – and voilà! Programmer didn’t think of it, but a user can do it quite naturally.

Submitting a query with a date or date range in order to receive a report is quite common functionality in many applications.  Here I want to share how analysing response messages helped guessing programmer’s logic and finding gaps.

As a first step, I figured that expected format is MM/DD/YYYY, and front-end JavaScript marks “invalid” an input which is not exactly 10 characters.  Then I immediately entered MM0DD0YYYY combination and received a response that either “/” or “-” must be used as a separator. Nice! As it turned out, front-end filter accepted MM-DD/YYYY, MM/DD-YYYY combinations, but SQL on the back-end generated an error. In another branch of ideas I tried M/DDD/YYYY, MM//D/YYYY, MM/ D/YYYY combinations, and figured that front-end validation again misses some of them, while in the back-end that led to exceptions. 

To admire programmers, the all exceptions were properly handled, so the application didn’t crush or broke business logic flow. However, my point was that properly filtering an invalid input in the first place is better behaviour than throwing back to user a message like “your input caused a problem but we’ve successfully recovered”, and development lead agreed upon that. 

I want to emphasize here that those described above were quick trial-and-error tests without any automation. Automated GUI tests were engaged later in exhaustive testing to throw hundreds of possible combinations of valid and invalid boundary data to every field, and log behaviour of the system.

2. “Broken Procedure” Tests

Most of user-computer interactions are based on linear procedures: “input-output”, “request-response”. Trying to distort a procedure in time or context sometimes brings interesting effects.

You may try the example, illustrated with screenshots below, on your system with Windows Explorer; the defect it causes is quite harmless and does not require any recovery steps.  

(Note. The consequences of human mistake triggered by the defect might be not so harmless.)

1. Find or create a folder with a few subfolders inside.

2. Double-click on the second last subfolder to get inside.

3. Click “Back” button on Explorer’s toolbar.
Now you are back in the parent folder. Note, that selection (highlighted list item) shows the last item you clicked, which is quite valid.

4. Now try selecting together the subfolder you’ve been inside (it was the second last one) and the last subfolder. If you tried on other subfolder, somewhere in the middle of the list, that’s OK. Still try to select a range between your subfolder and the last one.

While holding “Shift”, click on the last subfolder.

Oops! Somehow you get all subfolders selected.

What can we get from here? During implementation of the selection procedure the following was assumed:

• Group selection takes the all list items between first item clicked and second item clicked
• Clicked item, highlighted item, and selected item is the same

And, as it turned out, the second statement is not always true, but it can never be discovered if you test selection functionality as a separate and isolated procedure!

Now let me thank you for reading and you are more than welcome to share your own examples.