One of the basic tenets of unit testing is that test methods should be independent of each other.  JUnit goes to extremes to guarantee this principle by reinstantiating your test class before every method.  I personally think that as soon as you do more than unit testing, your test methods will invariably become dependent on each other (and as a matter of fact, what are setUp() and tearDown() if not dependent methods?), and that a testing framework needs to account for that.  But that’s not the point of this post.

Recently, I started wondering why this principle seemed to be so important and I asked the following question to several developers:  "Why is it important to have test methods that are independent of each other?".

I was quite surprised to receive pretty much only one type of answer:  "So you can rerun your tests easily".

My surprise comes from the fact that this is a tool concern and not a core design principle.

What was even more surprising is that JUnit doesn’t actually give you any way to achieve this task, but let me give you an example.

Every night, thousands of tests are run and when I come into the office in the morning, I check the reports and investigate the failures.  Being able to quickly rerun the failures is of critical importance to developers but there is no easy way to achieve this with JUnit.  Actually, there is not even an easy way to run a specific test method without the assistance of a third-party ant task, and at any rate, such add-ons are not really helping you with the problem at heart:  you should be able to rerun the failed tests the same way your nightly tests ran them.

Now, imagine a testing framework that would give you a very easy way to rerun only the tests that failed in the previous run.  Would independence of test methods from each other be so important any more?

With that in mind, I spent a couple of hours on the plane yesterday coding this feature for TestNG (you saw that coming, didn’t you?), and it turned out to be quite trivial.

TestNG runs tests based on a definition file called testng.xml.  Whenever tests fail in a test run, TestNG will create a corresponding testng-failed.xml that will contain only the tests that failed.  Therefore, a typical session will look like this:

testng -d output testng.xml
testng output\testng-failed.xml

This task is made slightly more complicated by the fact that TestNG supports dependent methods, so if a failed method depended on the successful run of previous test methods, these test methods must also be included in testng-failed.xml, but this was trivial to achieve since TestNG already knows the order in which the methods must be ordered.  It was just a matter of filtering out all the test methods that succeeded and/or were not necessary to rerun the failed tests.

I already started using this feature to debug TestNG itself and it’s made me save a lot of precious minutes already.