Object Definitions Blog

Well perhaps speed is not everything, but it is an important consideration when selecting a testing framework.

I was briefly brought in to help with a struggling project recently. In theory it was all good test driven development – that ought to improve the chances of somebody new to the project being able to set up a working development environment and make changes without breaking anything critical, since the tests are there to guard against any deliberate mistakes, right?

Well that is all good in theory, but in this case the unit and acceptance tests were taking well over three hours to run. To make things even worse, since the project was in trouble as I mentioned, there were some intermittent bugs lurking which had never been rooted out. This would mean that even after running for a couple of hours there was a good chance that the tests would fail in a manner which was not necessarily reproducible. Some of these bugs were probably associated with the testing framework code, rather than the actual product under test.

So what was the implication?

Well on any day, you could at most run the test suite twice, and even then the results could not be relied upon. For a new developer setting up an environment, it might take at least a couple of days just to achieve a first successful test run, or perhaps it might never be achievable. This totally negated all of the value normally associated with a comprehensive test suite, and actually made the tests worse than useless because of the time wasted waiting for the flawed tests suites to run.

As time went on, rather than spend time fixing the test framework and improving the speed, the team (because they were under pressure to fix the bugs, right?) opted to allow broken tests to stay broken, delaying their fixing to a single massive crunch in the days and hours immediately before each release.

This is probably not what the gurus of XP intended…

There are some lessons which can be learned here:

1. The tests are at least important as the product being tested, and should be maintained as such. Always fix tests immediately.
2. The testing framework is at least as important as the tests. Time spent improving it is time well spent!

And lastly, I have a new rule of thumb I am calling the long test suite conundrum:
The value of a test suite decreases as time taken to run it increases

Longer running test suites have lower value to a team. Of course many other factors affect the overall value as well, but it is pretty intuitive that a test suite that never completes has almost no practical value, whereas a test suite that can be run in seconds gives instant feedback, and therefore has high value. (This effect can be offset slightly by continuous integration environments which run the tests automatically, but the basic principle is still valid). Speed is therefore an issue of critical importance in a testing framework.

One of the side effects of the above rule (and the reason it merits the conundrum label!) is that by adding more tests you might actually be devaluing the test suite if your tests run slowly! So, even if you are improving the coverage, the overall effect may be detrimental, depending on how long the new tests take. So, it seems more tests are not necessarily a good thing. The ideal test suite has 100% coverage and executes instantly, whereas a suite of tests with zero coverage which never terminates is the worst possible. Either increasing the coverage or decreasing the run time will increase the practical value of a test suite to a team. So this is why it is even more critical that tests are optimized at least as much as production code, and that your team has tools which run tests quickly and efficiently. Simply adding more tests while overlooking optimization can have very bad long term consequences.

My current team recently rejected several Swing testing products since although they were rich in features, they increased our test suite time to well over an hour, from the current 5 minutes.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.