Parent page: Load/Performance Testing with NeoLoad

Providing irrelevant or misleading test results is worse than not providing test results at all. When using automation, testers must be twice as careful about relevance of tests – or maybe ten more times, hundred times – depending on how much judgment is passed on to mindless side.
Load testing is particularly challenging from this perspective, because tester is limited in using her native observation senses. It takes long practice to start “feeling” those problems that emerge only during concurrency and load, and it takes even longer practice to construct load profiles that will trigger the problems to emerge. With time, automation engineers add those heuristic tricks into their arsenal. Some of the tricks are very context-dependant, so it’s hard to share the ideas without disclosing the context. But some are quite generic, and can be used as a common sense advice.

“Stress is not a good way to start” – here’s the today’s heuristic (unless you’re intending to test behavior under spiking stress, which might be important to know, too, or you’re intending to explore how the system recovers after a heavy stress… but, that’s an idea for another blog post).

So, we need to run a constant load against the web site.

In NeoLoad, that would be a policy setting like this one:

The load coming from 0 to 100 concurrent connections in a very short period of time (it might be from seconds to minutes, depending on how many requests and how “heavy” they are) is an extreme stress for a regular web-site, and such stress will impact benchmarks drawing them irrelevant.

Now, different load testing tools have different capabilities for handling this problem.

Some tools don’t handle it at all.

· You can construct custom load profile by combining “Ramp-up” and constant load.
· You can track the point where web-site has recovered from the stress and manually remove a piece of measurements taken prior to that

 Some tools “cut out” measurements taken in the initial phase of load.

· You can track the point where web-site has recovered from the stress and properly configure the tool.

With NeoLoad, you can configure the load profile itself, by setting virtual user start policy.

By default, it’s simultaneous start.

Based on the number of requests,  and how “heavy” they are – investigate and apply your judgment here! – you can set a reasonable delay between starts of users, so that the web-server is not over-stressed.

To illustrate the difference, here are the graphs of two load profiles using exactly the same script, but with different starting policy.

The green graph is a response time (Y axis) over duration time (X axis) when the load is gradually increased.
The red graph is a response time (Y axis) over duration time (X axis) when the load is simultaneous.

Note that after a period of bouncing the red graph starts getting close to the green one – that means, the application is recovering from stress, and results are becoming relevant.