The Problem:

A unit test is failing when run as part of a test suite. When you run the test on its own, the test passes. The cause of this is that some state (a static variable or thread local) is being set some time earlier in the suite, and is not being torn down properly.

These bugs are hard to track down manually because they do not “fail fast”. The source of the error is some indeterminate number of tests before the error itself.

The Solution:

A useful technique I’ve developed for tracking these errors down uses JUnit test decorators. Test decorators allow you to attach additional set-up and tear-down behaviour to tests.

Step One: Isolate the Problem State

This is usually pretty easy to do just by examining the way in which the tests are failing. If something succeeds on its own but fails as part of the suite, working backwards to what data isn’t being cleaned up properly rarely presents much of a problem.

Step Two: Write your decorator

private static class LeakFinder extends TestSetup
{
public LeakFinder(Test test)
{
super(test);
}
public void tearDown() throws Exception
{
String testName = this.fTest.toString();
System.out.println(testName);
super.tearDown();
}
}

By the time the tearDown() method in your decorator is called, all the state in your test should have been cleaned up. This is where you put your test to ensure the state has been cleaned properly. For example, a common thing we lose track of in Confluence is the thread-local HttpServletRequest that we sometimes have to mock out:

public void tearDown() throws Exception
{
try
{
if (ServletActionContext.getRequest() != null)
throw new IllegalStateException("Servlet request not cleaned up: "
+ this.fTest.toString());
}
finally
{
// So that the error doesn't propagate to the next test
         ServletActionContext.setRequest(null);
}
super.tearDown();
}

Step Three: Decorate Your Tests

Next, you’re going to need a test suite that wraps all your tests in this decorator:

public class TestLeakDetector
{
public static Test suite()
{
TestSuite suite = new TestSuite();
suite.addTest(new LeakFinder(com.example.TestSomeStuff.class));
suite.addTest(new LeakFinder(com.example.TestOtherStuff.class));
// and so on for all your tests
        return suite;
}
}

Writing this test suite by hand is incredibly tedious. I use a three-line Ruby script to find all my test classes and write the suite class for me, but for a Java team it probably makes more sense to do it as an ant or Maven task, so it can be maintained with the rest of your build system.

(If all your tests share a common base-class, you might skip the decorator entirely, and just put the test in the base-class’s tearDown(), but I wouldn’t recommend it. Putting it in the superclass doesn’t detect places where some test has overridden the method and forgotten to call super.tearDown(), or tests that don’t extend that base class, and might also give false results for tests that call super.tearDown() out of order for some esoteric reason.)

Step Four: Run It

So now you have a test suite that runs each of your unit tests, and checks that they correctly clean up their state after themselves. Run your leak detector, find out which are the guilty tests, and fix their tearDown() methods to correctly clean up after themselves. While you’re at it, you might want to make sure they’re correctly cleaning up their instance variables so you’re not wasting memory.

A Better Solution

It would be nice if you could pass in the name of a decorator to JUnit’s test runner, and have JUnit automatically decorate each test without having to go through the suite-building step.

(Having gone all the way through writing this post, I discovered via Google, amusingly enough, that I wrote about the same thing in my own blog two years ago. 🙂 )

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now