This week I set out to add performance testing to a project I’ve been working on. Why performance test? The main benefit is that we can hone in on when one of the changes we’ve made to a project has affected performance dramatically(such as memory usage, run-time, etc.). It gives us the ability to review performance historically and subsequently identify areas of improvement.
The catch with performance testing is that run-time can vary between runs, making it tricky to test. There are a couple ways to tackle this problem:
The first of which is to run each performance test multiple times and average the results; then compare it to the previous run. This is the most accurate way to go about it, but not necessarily the most cost effective way to spend your time. For instance, if your performance tests take 30 minutes to complete and you need 10 runs to get a good average, that’s a 5 hour test.
Alternatively, we can compare the results of a single test to previous runs and simply identify whether or not the test falls within a desired percent of our distribution. This doesn’t have quite the accuracy of the first approach, but if you don’t expect extreme performance changes it can be a viable option. The biggest downside to this approach is that you won’t necessarily detect deviations immediately.
Right now we don’t have the time to run each test multiple times, so we will be implementing our tests using the second method. For testing the run-time I created a simple function that takes a lambda expression as an argument, so usage would be something like this:
