Not a Load Test Plan template
(An example of a situational feedback approach)
The Law of Large Numbers theorem states that the average of results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed. Unfortunately, application’s behavior under a load abides much more casual Murphy’s Law. To make it completely bad, in testing we have to consider worst happened case. That is, if 99 users (of 100) were successfully able to pay their bills online and only one got transaction lost once, the payment application is nowhere near 99% usable, and we have to catch somehow this error that didn’t occur during functional testing.
Load Testing - simulation of concurrent user activities and investigation of application’s behavior under such circumstances.
So, what would you want to test?
Do you want to verify everything?
- This might take twice as long time as your time before the release. Or maybe 10 times longer.
So, how could you plan load testing?
In my testing experience I had both projects when I was asked for specific answers (E.g. page response time) and where I was solely responsible for testing strategy and approach; when the all activities were planned and scheduled and where testing was performed ad hock and upon request. Based on those experiences, I suggest to assess questions ”What can you do?” and “What results will be of [immediate] value?” before designing a Load Test Plan. Answers to first question heavily depend on availability of Test Environment and testing tools (and, of course, skills of a tester). Answers to second question are context-specific: they depend on the project type and project phase.
If your Test Infrastructure is not as powerful as Production Infrastructure?
Importance of having Test Environment equally powerful (and with identical infrastructure) as Production Environment is critical. However, sometimes Test Environment is not always available, is not yet available, or just not available due to a variety of reasons.
What is worthless to do?
Any thresholds and bottlenecks found will not match with those on Production Infrastructure. Any back-end performance stats will not match with those on Production Infrastructure. If performance of some infrastructure components of the same type (for instance, database) is comparable then high volume tests or stress tests could give some relevant information but still very limited.
What can you do?
Front-End application performance is performance of a client-side part of it. If a web page is rendered slowly in a browser then it won’t matter how fast infrastructure is. So you can investigate front-end performance even with poor infrastructure and without generating a load.
Most of web-based applications are supposed to be accessible 24/7. On a back-end that requires endurance and robustness from application servers and databases. Endurance for a server means careful management of available resources and robustness means ability to recover after errors. If an application server side has problems with RAM management then memory leakage will eventually slowdown or crash it. More powerful hardware can only prolong up-time.
If you are a part of application development team?
Earlier detection of errors gives more time for fixing them. What kind of feedback on application’s performance is the most appreciated during development phase?
Same implementation approach is usually shared across the application. Any page performance issues, discovered early in development, might have an application design level positive impact.
Load/Performance testing of back-end components and entire infrastructure is often used with the main reference to a database performance testing. While many “bottlenecks” are truly often found within database, the following investigations should be conducted as well.
- An application server performance and behavior during different types of load (stress, spikes, etc.)
- How load balancing module affects performance?
- How anti DDOS module (or any Firewall) affects performance?
- How encryption affects performance?
- What level of tolerance has an application to latency, packet loss, link faults and other dynamic routing effects common for dial-up, ADSL, WAN types of connection?
Endurance and Robustness
Most of modern programming platforms have memory management functionalities hidden deep inside and not directly controlled by developers. Monitoring performance of application servers helps gathering information on efficiency of so called “Garbage Collectors”, “Thread Classes”, and other components of an application that do not carry business functionality but may have a direct impact on it. Any other resource allocation and management problems if discovered earlier will help saving time and increasing the potential performance of an application.
Once components of an application are linked and functional (usually, that is System Integration Testing phase) scalability tests will help to identify performance thresholds of an application. Test results will be meaningful if the Infrastructure -Under-Test (or, at least, its tested components) is similar to Production Environment, and load levels are equivalent to or exceed expected load in Production.
If the project goal is deployment of a vendor provided application?
You may still complain about defects found in off-the-shelf software, and they will eventually get fixed (most likely, in the next release) but implementation project can’t wait that long. It is critical to find out as much as possible about all instability thresholds, possible vulnerabilities – and test how well they could be covered by external measures, such as load balancing or filtering tools, expansion of hardware or network resources.
So, what tests should you target first?
Scalability and Performance Tests
If the infrastructure is already designed and implemented it’s really critical to find out whether its capable of carrying the expected load and what capacities it has. If any performance requirements are defined (or provided) by stakeholders you may conduct performance compliance tests. Otherwise you may capture performance characteristics at the highest level of load where application is still stable and provide stakeholders with that information. Those performance characteristics might also be used in regression/comparison testing whenever application or infrastructure change in the future.
Stress and Endurance Tests
Finding out how infrastructure behaves under extensive load or during peak periods is a next priority task. Issues found might not be fixed programmatically but still could be workarounded and definitely need to be identified. It also could be not a one time job if you’re helping in configuring of load balancing tools or firewall software.
I am a firm believer that the role of documentation is critical in testing. However, I value beforehand created documentation much less than structured records of how it actually was happening. I also often prefer digging extra time around issue(s) found rather then proceeding along with a plan. I always plan but my plans are essentially goals and milestones to reach rather than steps to walk. I prioritize them on the flow, based on real-time information received and events that have already occurred. This is a situational feedback approach.
Do you prefer Procedure Plans or Goal Planning?