Can you really have too many tests?

I’ve been working with an old-school client who operates the classic dev team/test team model. The dev team has become test-infected over the past 5 years, writing all flavours of automated test, from unit through to full blown story tests, with Junit, Selenium and the like. Unfortunately, the suite is quite flaky and even when it’s working now takes 17 hours to run. The test team, by contrast, have a huge body of manual test scripts that take weeks to run through, are of uncertain quality and have areas of overlap with the dev’s automated tests.

Some years ago, on one project, the dev and test teams tried working together to reduce duplicated test effort and reduce the reliance on manual testing. The devs talked through the Selenium tests they were writing and the testers were able to trust the results of the automated build. Over time, however, this trust eroded, partly because the test manager insisted in having complete regression coverage in the manual scripts, partly because the test team can’t read the Java-based tests and partly because the cooperation between the teams was an adhoc arrangement for a single project, and was never formalised within the organisation.

So, the perception within the organisation is that the dev team deliver very slowly and that the test team’s manual approach is an unnecessary burden – two major drags on delivery of business value. The project that I’m involved in has the stated goal of automating the regression suite, a classic technical response to what is essentially an organisational issue. Thankfully, there’s enough realisation within the organisation that working practices need to change to give some hope that any benefits that this project delivers won’t be lost with the next change request.

One of the inevitable side-effects of having a comprehensive test suite is that it will almost certainly be impractical to run all the tests all the time. Even if they were to adhere strictly to the testing pyramid distribution and all their unit tests were written according to the Feathers’ definition, it’s still going to take longer to run the full suite than is comfortable in a development or CI environment.

Recently Michael Feathers talked about time-boxed test suites and this is certainly what I’ve been recommending to clients for several years. A key thing to bear in mind when adopting or designing your test framework is the flexibility of the mechanism(s) provided for the selection of subsets of the full suite. Different suites will need to be run in different circumstances: a developer will want a suite to run locally in seconds, while a CI server can cope with minutes and an overnight build can take hours.

Cucumber (and now Cucumber-JVM) provide a powerful tagging mechanism that can help with this. A home grown system that we implemented at IBM had a similar mechanism. Crucially though, you should take the time upfront to think about the attributes you want to tag your tests with. Some aspects you might want to consider when designing your tagging lexicon include: speed of script execution; required dependencies for script to run; the area of functionality that the script verifies; the release that the feature was introduced; the defect fix that the script verifies.

Over time, your subsets will also grow. You’ll need to regularly re-evaluate which tests deliver the best value. Ideally you will take a risk based approach, moving older, stable tests from the ‘fast’ suites to the overnight or weekend builds. And like mowing the lawn, it’s much easier to do this job regularly, before the whole thing gets too overgrown.

Michael also suggested deleting some tests to reduce execution time. I don’t like this idea, even though, as he points out, the tests will still be available in your version control system. I think of the tests as the executable specification of the system and I don’t want to have to look through ‘deleted’ tests to determine its expected behaviour. Only delete tests if they have no further value – if they are still valuable then use your automation framework to define appropriate subsets that fit inside the time-box.






Leave a Reply

Your email address will not be published. Required fields are marked *