24Dec/09Off
Ideas for stress testing
With most project teams I've worked with, I've found that when I say the words "stress testing" people really hear the words "load testing." To me this is rather odd, but when I step back and think about it, I guess it makes sense. Most applications I've worked on have been web applications, and in that world load is often one of the biggest factors related to stress.
However, to help remind myself that there are other forms of application stress, I use the following checklist when I evaluate what stress testing I want to do. For each of these, I look at each piece of the architecture (desktop/browser, database, service/server, etc...) and in turn ask myself, "What would happen if...?"
- heavy load:
- large number of users
- large volumes of data
- large number of processes
- large number of transactions
- constrained resources
- not enough memory, processing power, or disk space
- not enough bandwidth
- not enough queues (or even shallow queue depth)
- not enough licenses
- inability to access files, servers, or services
- unfriendly inputs
- invalid file types and/or data types
- fuzz testing
- very large (or very small) inputs
- shared resources
- crash or stress other applications in a shared server space or that use a shared device/service
- timing
- constant or steady load for long periods of time (endurance)
- intermittent loads with long pauses in between transactions (looking for timeouts or parts of the system that go to sleep or get deprioritized with inactivity)
- concurrency (trying to force potential race conditions, file/database locking, etc...)
December 24th, 2009 - 13:06
Great work on the Daily Testing Tip. My compliments to the authors!
For me, load triggers ideas about heavy but plausible loads. The purpose of load testing is to see if the application stands up to normal, heavy, or extreme conditions, but conditions that are in some way expected and plausible. In that sense, “load testing” takes a kind of confimatory emphasis.
For me, stress has a different motivation, a more investigative goal. The idea is to try to push the application fail with open expectations. We overload it to the degree that we anticipate that it will fail, but we don’t know where, when, or how. If you like, load testing is about making sure that the chain will take the weight; stress testing is about finding out which link is going to break first.
The foundation of these ideas came to me from Cem Kaner.
When they think of “large”, some testers focus on boundaries. There are two pitfalls here. One is that they might look at “one more” than the stated boundary, rather than “many more”. The other problem lies in focusing on known or announced boundaries, where there are lots of boundaries that might be hidden, dynamic, unknown, unrecognized, mis-stated, and so forth. So rather than thinking simply of “large”, I’ve found it helpful to think of “way too much”, “way too many”, or “”outrageous”.
One more thing: in addition to devices or services, data is often shared. Running multiple instances of the same application, or running several applications that have access to the same data, can expose important data integrity-related bugs.
Cheers,
—Michael B.
December 24th, 2009 - 13:28
Thank you Michael, I appreciate the compliment. And thanks for adding data to the list of shared resources. You’re right. I’ve worked on several projects where I’ve seen dynamic reference data cause issues.
-Mike