I created this mindmap quite a while ago, as a follow-up to Data Container Boundaries and Complex Data Boundaries blog posts. Due to poor visualization capabilities of Freemind, I used previously, (before going with XMind), I decided to draw the mindmap manually, in MS Paint. That took me loooong time but I liked the result. Although, now I would write the posts differently in some aspects, and I will get back to the theme of heuristic-based boundary testing.

Below you can view and compare manually created map and the one designed with XMind tool.

While getting ready for the Rapid Software Testing course I dedicated some time for the “homework” – studying of publicly available exploratory testing documents. And here’s my ideas on Functionality Testing.

References

Heuristic Test Strategy Model – Satisfice, Inc. – Designed by James Bach

Michael Bolton, DevelopSense: Testers Know That Things Can Be Different

While getting ready for the Rapid Software Testing course I dedicated some time for the “homework” – studying of publicly available exploratory testing documents. And here’s my ideas on Claims Testing.

References

Heuristic Test Strategy Model – Satisfice, Inc. – Designed by James Bach

Michael Bolton, DevelopSense: Testers Know That Things Can Be Different

How to find bugs quickly? How to pick just right test scenarios without losing your time trying thousands of possible combinations?

These questions worth years of research.

In my post I’m going to cover just one simple approach I practice quite successfully, and give a few examples you might be able to reproduce.

Let’s start from definition.

Probability is a way of expressing knowledge or belief that an event will occur or has occurred. The concept has been given an exact mathematical meaning in probability theory, which is used extensively in such areas of study as mathematics, statistics, finance, gambling, science, and philosophy to draw conclusions about the likelihood of potential events and the underlying mechanics of complex systems.
Wikipedia.org

As you can see, almost everything from the definition above applies to software testing. Complex systems – that’s what we investigate. Mathematics and statistics - is what we use in analysis. Gambling? – Well, testing involves guessing, and the thrill of the bug hunt is for sure addictive. Overall, software testing is an alloy of science and philosophy.

Considering probability

Exhaustive investigation of all possible boundary conditions is very time-consuming.  In the same time, after any code change of the particular functionality all boundary investigation results become obsolete. That is, unless you have infinite time, the primary goal is not to find all boundary bugs but look until we find the first important one, and then move on to another piece of functionality (Why? Please read about “The Dead Horse” and other stopping heuristics).

And here is the rule of thumb I use.

Try first those Test Scenarios that

least likely were considered during implementation
and/or
much likely can happen during use of the system.

 Note, that these two probabilities are not the same number. It could be “not considered but much likely to happen” (worst case) or “surely considered and unlikely to happen” (happy case).

Examples

1. User Input Filtering and Validation

Let’s assume, there is a requirement: a user can enter only numbers in the text box.

You try keystrokes using every single button on the keyboard, and see that invalid characters are successfully filtered. Then you type an invalid combination in Notepad, Ctrl-C, Ctrl-V, – and voilà! Programmer didn’t think of it, but a user can do it quite naturally.

Submitting a query with a date or date range in order to receive a report is quite common functionality in many applications.  Here I want to share how analysing response messages helped guessing programmer’s logic and finding gaps.

As a first step, I figured that expected format is MM/DD/YYYY, and front-end JavaScript marks “invalid” an input which is not exactly 10 characters.  Then I immediately entered MM0DD0YYYY combination and received a response that either “/” or “-” must be used as a separator. Nice! As it turned out, front-end filter accepted MM-DD/YYYY, MM/DD-YYYY combinations, but SQL on the back-end generated an error. In another branch of ideas I tried M/DDD/YYYY, MM//D/YYYY, MM/ D/YYYY combinations, and figured that front-end validation again misses some of them, while in the back-end that led to exceptions. 

To admire programmers, the all exceptions were properly handled, so the application didn’t crush or broke business logic flow. However, my point was that properly filtering an invalid input in the first place is better behaviour than throwing back to user a message like “your input caused a problem but we’ve successfully recovered”, and development lead agreed upon that. 

I want to emphasize here that those described above were quick trial-and-error tests without any automation. Automated GUI tests were engaged later in exhaustive testing to throw hundreds of possible combinations of valid and invalid boundary data to every field, and log behaviour of the system.

2. “Broken Procedure” Tests

Most of user-computer interactions are based on linear procedures: “input-output”, “request-response”. Trying to distort a procedure in time or context sometimes brings interesting effects.

You may try the example, illustrated with screenshots below, on your system with Windows Explorer; the defect it causes is quite harmless and does not require any recovery steps.  

(Note. The consequences of human mistake triggered by the defect might be not so harmless.)

1. Find or create a folder with a few subfolders inside.

2. Double-click on the second last subfolder to get inside.

3. Click “Back” button on Explorer’s toolbar.
Now you are back in the parent folder. Note, that selection (highlighted list item) shows the last item you clicked, which is quite valid.

4. Now try selecting together the subfolder you’ve been inside (it was the second last one) and the last subfolder. If you tried on other subfolder, somewhere in the middle of the list, that’s OK. Still try to select a range between your subfolder and the last one.

While holding “Shift”, click on the last subfolder.

Oops! Somehow you get all subfolders selected.

What can we get from here? During implementation of the selection procedure the following was assumed:

• Group selection takes the all list items between first item clicked and second item clicked
• Clicked item, highlighted item, and selected item is the same

And, as it turned out, the second statement is not always true, but it can never be discovered if you test selection functionality as a separate and isolated procedure!

Now let me thank you for reading and you are more than welcome to share your own examples.

Image courtesy: starwars.wikia.com

You must unlearn what you have learned.
Master Yoda to Luke Skywalker

Although Unlearning is not a testing heuristic itself, I found it very helpful in many activities – from testing and test automation to problem solving and management. That is why I wrote about Unlearning on quicktestingtips.com.

Yet I wanted to share even more about Unlearning, applied to testing, and beyond that, so here is my today’s post.

Test Requirements

It’s nice when you have detailed test requirements which instruct you to verify that an application works the way it meant to do. But keep in mind that users don’t always read guides and instructions (or might even say – never read), so unlearn “right” user habits and try to do same tests in a weird manner.

Test Management

There is a long time known IT joke saying “Project Manager is a person who thinks nine women can deliver a baby in one month”. The idea of replacing experienced staff with cheap, unskilled resources, supplied in great numbers, with a hope that the same results will be achieved at low costs, still lives. However, if you want to unlearn it (or convince someone), I suggest you reading “The Taylor Termination Programme” story published on polite-unlearning.org web-site.

People were taught to believe in numbers. If your approach is “metrics first”, you may want to know that you will die from tomatoes. Surprised? Read the following case study – “The Dread Tomato Addiction“.

Testing

Unlearning and re-learning testing habits could be fun.

Used to navigate through an application with mouse only? Try keyboard shortcuts, Space/Enter instead of clicking, Cut/Paste instead of typing, holding Ctrl/Shift during GUI manipulation.
(One of little tricks here is that modern software has a lot of functionalities built-in by default. Operating in an unusual way you trigger them, or you not trigger pre-programmed event handlers thus affecting inner flags and counters in a program.)

Breaking sequences and not following assumptions is another way. Try performing an action before or after it’s expected, double-click instead of single click, manually pasting urls instead of clicking navigation links, Undo/Make changes/Redo cycles. Rename binary file to txt extension and feed it into your application. Change XML tags. Try French or Chinese input instead of English characters.

Parallel or reversed work instead of sequential. Here we have a variety of options; below I gave just a few examples that helped me exposing bugs in the past.

• Save / submit data (file, report, web-page, …) while requesting it as another user
• Send / request data and disconnect session before receiving a confirmation)
• Delete (to Recycle Bin) a named item (file, web content element, etc.), create a new one with the same name, and try to restore the deleted item
• For scheduling and filtering functionalities, change computer dates and item dates

Test Automation

A successful test automation engineer must be proficient both in testing and programming.
However, design of testing scripts requires unlearning from both tester’s and programmer’s habits.

That is, the following should be unlearned:

• automated testing is execution of testing scripts
• testing script should mimic tester’s actions
• testing script is a program that drives another program

Further reading

Unlearning101 by Jack Uldrich

Blink testing, as described by James Bach and explained by Michael Bolton, is an oracle heuristic based on “snap judgment” effect.

Try catching “bug” (an odd character) in patterns below.
I arranged them by difficulty I had. Feel free to put your experience in comments.

In full-screen view all characters are 10pt capitals, as in standard Notepad window. 

Easy difficulty

(click to see full-screen in a separate window)

(click to see full-screen in a separate window)

Medium difficulty

(click to see full-screen in a separate window)

(click to see full-screen in a separate window)

Hard difficulty

(click to see full-screen in a separate window)

Impossible difficulty

(click to see full-screen in a separate window)

Copyright notice

I maintain no copyright on the idea of such puzzles or the patterns used. I saw many of them on the Web on free resources, and I received some of them as fun stuff via mail.
What I did was creating the patterns of same size, and saving them as images, to prevent cheating through source code view.

Complex Data Boundaries: overflow or type mismatch?

Complex data types are created by composition of basic data types. Compositions of data of the same type are formed as arrays. Compositions of data of different type are formed as records.

There are internal (technical) rules defining how complex data types are managed. The rules are platform and programming language specific. 

Arrays and pointers

For the each dimension it has, an array has minimal and maximal index boundaries defined as numbers. In some programming languages array index begins with 0, in some other – with 1. That leads to lots of confuses based on:

• First element should be accessed as Array(0),
• Last element should be accessed as Array(Number_of_elements minus one) 

When reported to the screen indexes should be converted to 1-based notations. When an input is received it should be converted to 0-based index before addressing an array element. 

To overcome that issue programmers use various approaches. For example, I have seen the code below on a few projects and in different companies:

Months = Array (“Zerourary”, “January”, “February”, “March”, “April”, “May”, “June”, “July”, “August”, “September”, “October”, “November”, “December”)
“Zerourary” month is defined to occupy the first slot which has index 0. 

Arrays may occupy a static (defined) memory amount or dynamically allocated memory. Dynamically allocated memory is addressed by a pointer data type.

If a memory allocation operation fails, NULL (empty) value is assigned to a pointer. Any attempt to address memory under the NULL-pointer will cause a run-time error.

If memory is constantly being allocated and not released, a program may exhaust system resources and cause heap overflow error (and most likely will get crushed).

If memory was released then the pointer has an uncertain value (“dangling pointer”). If program code tries to use it (address memory for reading or writing) it will cause an error and may result in a crush of the program.

Strings

String is an array consisting of characters. Some programming languages reserve 256 elements for a string type and store the actual number of characters at index 0 element. So the string may have up to 255 characters total. Depending on the programming language and compiler directives, when the actual number of characters is exceeded the string is silently truncated or an automatic exception is thrown. 

Enumerations and lists

Enumerations and lists define possible values that could be taken. In addition to index boundaries there are could be additional rules as in the following examples:

• an array must contain same or less number of items as in the enumeration (e.g. reporting week days in timesheet)
• an array may not contain duplicated items from the enumeration (e.g. week days should not be duplicated in the example above)
• some items may not be duplicated and some could be (e.g. one can have multiple brothers, sisters, sons, daughters but only one mother and father. To make it more complicated: some countries allow having multiple spouses) 

Records 

Records may represent closely related values, like year, month, and day for the date; or somewhat related only business-wise, for example last name and SIN. Often values stored in a record require transformation before output to the screen. These transformation rules are subject to change depending on regional settings and customization settings in the program. 

A code operating records is oriented to the specific format. Code modules that pass in and back data records should either follow the same convention with a format or have bridging functions that perform the conversion. That applies both to back-end and front-end functions. 

Testing the boundaries 

The approach 

While defects with these types of boundaries usually impose a high severity issues they can be relatively easy caught at very early stages of development. Implementation of automated Unit Tests is better investment here rather than implementation of automated functional tests.

However there are a few good functional test sets that allow quickly check the most common scenarios. Since complex data type boundaries are internal the main trick here is to make a code make a mistake. 

Examples. 

• Strings. Although GUI may not allow user to input strings exceeding the boundaries tester may try to make an application to produce a string of an inappropriate length. For example, if full name is constructed by an application as a combination like “last name comma first name” based on the manual input of first and last name.
• Arrays and enumerations. Tries to make a code assigning a wrong value work very well with combo boxes (leave black, try multi-select, try typing value not presented in the list), list boxes (select none, select all), radio groups (select none) and combinations of those, for example, try input two mothers. 

The challenges 

The challenges in identifying and proving a defect are similar to the ones described in data container boundary testing.

1. No immediate visible reaction

An application might accept an (invalid) input and still be alright. Causing it to output the value or use the value in some operations might help revealing a problem.

Boundary testing for a single value might require a whole scenario: post transaction – run processing – verify account balance.

 2. Internal mess-up

A single value might “travel” a pretty complicated path while being passed from one function to another inside the program. If somewhere within that chain the value won’t fit the memory cell the end-result will be wrong despite of the application accepted input. Or value could be properly stored in memory but will be messed up during saving into file or database record.

Test scenarios might be required to reveal those defects.

Data Container Boundaries: where is the weakness?

All the values used by computer programs are classified by type (integer number, floating point number, string, date, etc.), and all data types are internally encoded with numbers. Digital nature of computers makes them specific and discrete with numbers; once-and-forever defined memory cells serve that purpose.

The capacity of a memory cell defines boundaries of a range while the range itself is flexible. 

Examples. 

• One Byte data container – Byte, Unsigned Char (0  … 255), Signed Char (-128 … 127).
• Two Bytes data container – Word, Unsigned Short Integer (0 … 65535), Signed Short Integer (-32,768 … 32,767).
• Four Bytes data container – Double Word, Unsigned Integer (0 … 4,294,967,295), Signed Integer (-2,147,483,648 … 2,147,483,647). 

Note that an atomic part of a memory cell (a single bit) is used to encode sign of a number. 

Generally, memory cells that are to be allocated for storing values are explicitly defined in a program code by a developer (in case of compiler type programming languages) or defined by default settings (in case of interpreter type programming languages).

Now, what happens when a number doesn’t fit the boundaries? 

1. Native machine code behavior: arithmetic overflow

Note. It won’t stop CPU from further execution of machine code. 

Result. The memory cell will contain an uncertain but definitely wrong value. 

Consequences. Any further operations with the value from the cell will produce wrong results. If further execution uses the value from the cell as a condition of some sort then the decision made most likely will be wrong. 

An arithmetic overflow destroyed Ariane 5 space rocket (Flight 501). 

2. Language-specific machine code behavior: wrapping around or truncation

Note. It won’t stop CPU from further execution of machine code. 

Result.

• In case of wrapping around the memory cell will contain a value within the initial boundaries and relevant to the original one but definitely wrong.

Example. If Unsigned Short Integer data container is defined for the Amount on User  Account, and the Balance is 65K, deposit of 1,000 will change the Balance to 465 – which won’t bring down the program but might bring down the bank. 

• In case of truncation the memory cell will always contain a maximum allowed number. In other words, 10 + 1 will be 10 if truncation is in effect and 10 is the top of the range. 

Consequences. Any further operations with the value from the cell will produce wrong results. If further execution uses the value from the cell as a condition of some sort then the decision made most likely will be wrong. If the value was supposed to be used for memory allocation purposes or memory addressing purposes the entire operating system could potentially be brought down. 

Integer overflow vulnerability is widely (and quite effectively) used in remote attacks affecting both UNIX and Windows operating systems. 

3. System exception handling behavior: recovery

Note. System exception handling prevents the wrong operation to be executed in machine code. 

Result. Typically, a program generates an error report in a log file, shows to a user some kind of error message and finishes execution. More advanced programs can recover by partially or completely reloading themselves. 

Consequences. Typically, the current working session gets closed and all the unsaved data are lost. Server application might suffer from memory leakage, file/disk inconsistencies, or database inconsistencies. Closed Client side application loses a connection to server that might be overtaken. 

Generally, system exception handling scenarios are automatically added into program at the compilation time. 

4. Business logic behavior: avoidance

Note. In accordance to business logic defined, an application avoids erroneous situations by rejecting wrong data or handling those situations following programmatically defined scenarios. 

Result.

• In case of data rejection an application typically remains at the same state as it was.
• In case of error-handling scenario an application “rolls back”, typically to a preceding  state (or to an idle state), and generates some kind of report. 

Consequences. Typically, business logic based error-handling is desired and expected behavior of an application so there is no negative impact. 

Testing the boundaries

The problem

While all the data container boundaries could be easily found in technical documentation, identifying those boundaries is always a task of great challenge for tester, especially if design specification is not available (or simply there are no design rules mandatory for all developers on the project).

Another typical reason is a drift of business requirements. For example, originally maximum transaction amount was specified as 100 and then it’s changed to 1,000. Here two boundaries were crossed: 127 (Signed Char) and 255 (Unsigned Char).

Nowadays, different wizards, code generators, as well as external libraries and frameworks, bring even more chaos as developers lose control over the code, and may not be aware of data container boundaries.

The challenges

1. Search for upper boundary

Business requirements and common sense might help identifying the “starting point” of the search. If that can’t help then trying a “magical” numbers, like 128, 256, 32768, 65536 might help.

Using binary search methods we can get to the threshold faster.

2. No immediate visible reaction

An application might accept an (invalid) input and still be alright. Causing it to output the value or use the value in some operations might help revealing a problem.

Boundary testing for a single value might require a whole scenario: post transaction – run processing – verify account balance.

3. Internal transition

A single value may “travel” a pretty complicated path while being passed from one function to another inside the program. If somewhere within that chain the value won’t fit the memory cell the end-result will be wrong despite of the application accepted input. Or value could be properly stored in memory but will be messed up during saving into file or database record.

Test scenarios might be required to reveal those defects. 

Conclusion

There are two main weaknesses in digital encoding technology: data container overflow (memory cell overflow, register overflow) and a confusion (type mismatch) during conversion. Programs have multiple layers of defense to protect these weaknesses. Sometimes protection functionalities interfere with business functionalities.

If you know that protection is missing something you can find a method to expose a defect.

Software Testing is an empirical process.
Empirical approach as an opposite to theoretical requires
observation, investigation and experimenting.
Results of those actions performed are collected,
documented, analyzed, and categorized, before conclusion is made.

A Scientific Method of Learning consists of the collection of data through observation and experimentation, and the formulation and testing of hypotheses. If hypothesis is consistently confirmed in series of reproducible tests then it is considered truth. All the data, ideas, and methodology should be documented and shared so they could be objectively reviewed by side observers in order to verify results by attempting to reproduce them.

Everyone familiar with software testing and its role in Software Development will see a close match between it and Scientific Method.
There is, however, one major difference. Scientists research physical laws, that never change and apply everywhere in the Universe. For testers, every new program build is a new “universe”. We have to research its qualities all over again.

Investigating a computer program involves two kinds of processes:

• process of verification of known functionalities to see if they meet defined business and technical requirements;
• process of investigation of unknown functionalities and side effects in order to reveal their qualities, influence to and interference with defined functionalities.

The problem here is that known functionalities are almost never defined at the finest granularity level. There are always gaps and generalizations that filled with developers’ assumptions or even left missing. And even if requirements are defined at the best possible precision level there is always possibility of miscommunication or misunderstanding. As if that’s not enough, even in ideal situation if it’s coded exactly as specified there is a good chance that it will be messed up by the next release. Ironically, the current release is also still at risk because computer programs are sensitive to hardware and OS configuration, and dependant on their settings.  So if a test is passed in a test environment it is not 100% guarantee that the tested functionality will work in production. 

Investigation based methods are in better place as they do not rely on expected results but objectively provide information on program’s behavior. Furthermore, decades of worldwide practice demonstrated that investigation-oriented approach results in better software product’s quality rather that verification-oriented.  

Another good thing about empirical methods that you don’t have to spend (actually, waste) time and efforts creating Universal Laws. If something under some circumstances works more often than it doesn’t – you can use this approach. The approach may evolve as context and methods change; it can get abandoned and revived back. Any practical approach is a heuristic. With regards to software testing, I also want to link this definition to James Bach’s explanation.

Software testing heuristics are not bound to “Black Box” testing methods.

In the subsequent posts under this category I’m tracing the roots of different problems in software (bugs) in order to find out what are that weak spots and how they appeared.

Publications and References

“Testing Computer Software”, Cem Kaner, Jack Falk, Hung Q. Nguyen

“Lessons Learned in Software Testing”, Cem Kaner, James Bach, Bret Pettichord

Heuristic Risk-Based Testing by James Bach http://www.satisfice.com/articles/hrbt.pdf

Test Heuristics Cheat Sheet by Elisabeth Hendrickson http://testobsessed.com/2007/02/19/test-heuristics-cheat-sheet/

A challenge

Requirements are never complete or detailed enough… It might be a risk we have to consider or a challenge we need to take. This applies to anything in our job: from testing to development tasks.

How do I handle this?

An approach

The approach I use I usually call a breakdown or decomposition. One might call it a testing heuristic, but I would refer to it as a thinking style I gained after learning and practicing systems approach.

• Think of any entity as a component of a system (or multiple systems). Think of qualities that the enitity adds to the systems. Think of types of bonds that connect the enitity with other elements of the systems.
• Look inside the entity and learn about elements and bonds that form it. Learn what qualities have those elements, and what new qualities they create being formed into an entity.
• Look inside the elements that form the entity, and look upside the system(s) which part the entity is.