One of the biggest reasons I see performance testing fail is the non scientific approach to testing that usually ensues when something strange happens. Most of the time it causes a cascade effect in the test cycle - further tests get more crazy until the result you're looking at makes no sense, and trying to connect how you got to the last point is impossible.
Unfortunately, doing the right thing takes a lot more time. It requires a lot of patience and control to not jump ahead, and usually a lot of data collection.
Here are some tips I've figured out over time that help when I test performance. You may notice that it is similar to what a scientist might record when doing an experiment (albeit much simpler) - that's the analogy I use when explaining why performance testing is a difficult job, and time consuming.
1. Outline - come up with a general outline of where you think your testing will go. It doesn't have to be exhaustive, but it will help you figure out how to approach things. It will be really difficult to do #4 below (one variable change at a time!) if you don't have a clear plan as to what you're trying to prove/find/fix and the areas of the product/device you're suspecting things will be a problem.
2. Write it down - accurate notes of what you are testing, how things were configured, and the trail is critical in figuring out which result is the most useful. Try to use the test tool if possible to make notes about each individual test. Be as verbose as possible. My rule of thumb is that you should spend at least 5 minutes after each test noting the results in some format that will allow you to understand everything you tested a year later when you need to recall the information.
Saving the test data is critical (see below) but your observations and notes are even more important so you can remember why things happened the way they did, and what you observed.
3. Keep all your data - if your test tool supports it, every time you make a change, change the name of the test, or note the change you made. If it doesn't support it, figure out a way to combine tip #2 above with this so that you can accurately reproduce the test later.
I go even further and have a script that prompts me for my notes, gathers the configuration from the devices I'm testing with, and the device I'm testing, and package it all up in a simple tarball for use in the future. Someone could take this entire bundle and completely reproduce what I did, along with reading my general notes on the test.
4. One variable at a time! - this is probably the most important tip of all - only change one variable at a time - anywhere in your test environment. Many people will try this for a while, and when the result doesn't look good, or as the steps above get mundane, multiple things are changed, and then the trail can't be reproduced.
Often I have talked with folks about not changing too many things, and they aren't able to do it because of time restrictions. If that is the case, there are ways to optimize the above (like a script) that can make it still workable to take it a step at a time.
One of the best ways to make all of the above happen is to be in complete control of all of your equipment. Much of the testing I've seen that fails is when there are too many cooks in the kitchen - one person running the test gear, one person watching the device under test, and each of them watching differing things, and possibly making changes that don't match.
I'm sure there are a lot of other tips that folks could suggest, but these are the four biggest ones I've tried to stick to always, with good results.
Good luck!
Recent Comments