February 08, 2004

JUnit pain

JUnit

Why does a TestCase get instantiated as many times as it contains test methods?

Try this:

public class MyTest extends TestCase {

  public MyTest(String name) {
    super(name);
    System.out.println("Created a MyTest");
  }

  public void testA() {}

  public void testB() {}
}
and you will see "Created a MyTest" displayed twice.

My initial reaction when I see code that puzzles me is to give the author the benefit of the doubt, so my first thought was "Maybe they do this to make sure that test methods are independent from each other, that is, a test cannot fail because of a side effect created by another test method".

Then I realized that this justification was stupid for two reasons:

  • If I want two test methods to be truly independent, I will just put them in two different classes.
  • The setUp() method is called before each test method, thus guaranteeing that the state of the object is restored before each invocation.
The more I thought about it, the more I realized that being able to share some state between test methods, such as a JDBC connection or a parsed XML file, was actually quite useful. The problem is that JUnit makes this simply impossible:  you have to put this code in setUp() and therefore, rerun it before every test method.

At this point, and after finding all these deficiencies in JUnit's design and implementation, I am beginning to think we are once more stuck with a de facto standard that is utterly broken.

Unless someone can explain to me this behavior?

Posted by cedric at February 8, 2004 05:45 PM

Comments

Whether or not it makes sense, if you want once-per-test-class initialization, you can implement on your test class.

public static Test suite(){

TestSuite suite = new TestSuite(MyTestClass.class);

return new TestSetup(suite) {

protected void setUp() {
// one-time setup
}

protected void tearDown() {
// one-time teardown
}
};

}

Posted by: Tim Downey at February 9, 2004 09:35 AM

Why is this a problem? So what if it ititializes test# times?

Posted by: at February 9, 2004 10:47 AM

It creates unnecessary instances and can be very time consuming.

Posted by: Cedric at February 9, 2004 10:47 AM

Hi Cedric. I think a lot of it may just be defensive programming. You're right that you can create fixtures such as setUp() and teadDown(), but they don't have mandate the use of those fixture methods. So I think JUnit fires up x number of instances of the class if there are x number of testxx() methods and calls setup/teardown on each distinct instance and executes each of the testxx() methods separately. I think the goal is probably to create the cleanest possible test scenario and not worry about extra object creation and instantiation time.

Posted by: Vinny Carpenter at February 9, 2004 01:08 PM

The reason is that the running of one test should never affect the running of another. That was the design decision behind it, I believe - they want them isolated.

So if one test breaks, it won't cause another test to break simply by virtue of it breaking, and perhaps leaving some instance data in a bad state. Likewise, since they are independent you can select any subset and run that subset in any order. (Hmm, handy functionality that Eclipse doesn't seem to offer. Darn.)

Hence the number of class instantiations.

So hints Kent Beck in "Test-Driven Development by Example"

Posted by: Jon Mountjoy at February 9, 2004 02:13 PM

Indeed, this behaviour is just unexplainable to me - if the reason for the multiple instances is to avoid having stale data from failing tests hurting other sane ones, then the setUp and tearDown mechanism sounds quite a bit redundant to me...

Ok, being pragmatic: anyone tried Artima TestRunner?

Posted by: Carlos Villela at February 9, 2004 03:17 PM

In the JUnit FAQ they do describe how to do a one-time setup by using a TestSetup wrapper. I've done it, works ok.

See: How can I run setUp() and tearDown() code once for all of my tests?

J. Betancourt

Posted by: J. Betancourt at February 9, 2004 03:46 PM

It makes a little more sense if you look at how tests are used in a data structure. A TestSuite is basically a list of Test instances. Running a test suite looks like this:

for test in TestSuite:
  test.setUp()
  test.run()
  test.tearDown()

Since a test suite can contain any number of tests in any order, and the tests need not be instances of the same class, it probably wasn't immediately obvious how to optimize this.

I got around it with some custom subclasses: LazyTest and LazyTestSuite. A LazyTestSuite checks to see if two consecutive LazyTests have the same setup, and if so, calls the test's clean() in the middle instead of tearDown()/setUp().

Posted by: Brian Slesinsky at February 9, 2004 10:20 PM

I've also been finding junit (integrated with ant) to be quite frustrating. I wasn't aware of the new class for each test quirk, but I found that the ant batchtest created a separate TestSuite for each class and ran each in a different classloader. This meant that my Hibernate (and other) config would need to load many times throughout my tests. At around 4s per load, this was really annoying.

My solution was to create a DynamicTestSuite class that creates a single testsuite by matching a classname suffix that you give it.

E.g. Give it "Test" to match all test classes, "LgoinActionTest" for a specific class, or "ActionTest" to test all actions.

More detail (including source code) at: http://marc.theaimsgroup.com/?l=ant-user&m=107645600632538&w=2

Posted by: Joel Hockey at February 10, 2004 04:04 PM

I'm surprised the TDD fanatics haven't proposed putting you up against a wall and shooting you for daring to suggest that in the real world you might sometimes need fixtures that remain in place across tests.

Posted by: Hani Suleiman at February 10, 2004 06:57 PM

Cedric are you sure you are not confusing unit tests with system tests?

would not maintaining states or connections or objects between tests denote a system test insteadof a unit test?

Posted by: Fred Grott at February 11, 2004 08:06 AM

"if the reason for the multiple instances is to avoid having stale data from failing tests hurting other sane ones, then the setUp and tearDown mechanism sounds quite a bit redundant to me..."

If test cases shared the same fixture state and something failed during the tearDown of one test, the fixture would be left in an inconsistent state, and would cause confusing errors in subsequent tests running with the same fixture.

The alternative would be to require extremely defensive programming in the tearDown method, but that is impractical in practice: if you know where you should be defensive against programmer errors (RuntimeExceptions, for example) then you should just fix those errors!

Posted by: NatPryce at February 11, 2004 09:25 AM

A lot of people seem to either be talking around Cedric's problem, or simply apologizing for (as in, rationalizing) the behavior of JUnit, which I find to be a bit of a shame.

I'll attempt a different tack, and see if perhaps this illuminates the problem for the folks that don't see Cedric's point.

When you write tests, you subclass a class called TestCase. Not TestCases, not SetOfTestCases... TestCase, as in One Test Case. My expectation would be that an instanceof TestCase constitutes an indivisible unit of testability, and therefore should only need one set up and one tear down method call.

This notion that any method begining with 'test' in its name constituted a single unit test is excessively foreign to Java, despite the fact that numerous programmers seem to think it's a good idea to push things in that direction. The unit of design contract in Java is the Interface. Use it, instead of fighting to make the method name the unit of contract.

Posted by: Jason Marshall at February 11, 2004 12:32 PM

When I absolutely MUST share state between invocations of different test* methods in a test class, the solution is trivial:

public class MyTest extends TestCase {
static {
// shared setup
}
//...
}
the class only gets loaded once, so the static block is only run once.

Of course, as has been brought up many times, it's best to avoid doing so. But when you need to do it, it's easy.

Posted by: Jim Moore at February 12, 2004 05:48 AM

I've always wondered this.

I've thought for a long time that JUnit design is v. poor - hard to stomouch considering its ubercool parentage, and that orginal design paper on its evolution.

Two vaguely relevant mini-rants:

1. I so frustrated with the ANT JUnit task having to fork a process for every testcase (as in class, sigh - Jason is so right about the stupid naming; there are two types of sets of test cases, test suites of test cases which are actually Java classes of things that you and I would call test cases, and all of these are tests. got that?), because forking a JVM is very slow when you've got clearcase hosting half your classpath, that I contributed a version that didn't. I'm found I wasn't the only one to find that the ANT development community is deaf to such useful contributions.

2. The previous version of The Grinder (The Grinder 2 - http://grinder.sourceforge.net) has a JUnit plugin. Once upon a time I wrote this simple monstrosity to produce a list of JUnit tests decorated with my wrapper:

private void getTests(junit.framework.Test test, Set tests) {
// Really hacky switch on type, but no obvious other way of
// doing this with the JUnit API.
if (test instanceof TestSuite) {
final Enumeration jUnitTestEnumeration = ((TestSuite)test).tests();

while (jUnitTestEnumeration.hasMoreElements()) {
junit.framework.Test jUnitTest = (junit.framework.Test)jUnitTestEnumeration.nextElement();

getTests(jUnitTest, tests); // Recurse.
}
}
else if (test instanceof TestCase) {
tests.add(new TestWrapper((TestCase)test, ...));
}
else { /* Handle unknown type. */ }
}

which both sucks and blows. I don't think this was totally my fault.

Posted by: Phil Aston at February 19, 2004 11:28 AM

This problem is an absolute PITA to me and I had to do disgusting things to work around it in the Hibernate test suite (the initialization I need to perform is to read the mapping documents, create the SessionFactory, and export the db schema). The workaround? Keep the SessionFactory in a static variable and lazily initialize it from setUp(). Uggghhhh.....

Posted by: Gavin at February 22, 2004 09:31 PM

Learn Linux.

Posted by: Catherine at May 8, 2004 05:09 AM

For people complaining about Ant JUnit task's behavior..in Ant 1.6.2 there has been a new forkmode attribute introduced. More on this in the documentation: http://ant.apache.org/manual/OptionalTasks/junit.html

Posted by: Stephane Bailliez at July 26, 2004 12:49 AM

Ciao

Posted by: Sarah S at November 6, 2004 01:46 PM

I thought jim moore a simple answer:

public class MyTest extends TestCase {
static {
// shared setup
}
//...
}

and then I quickly realized that doesn't allow you to tear down the type of things you actually want to set up, like database connections.

Cedric is right. Junit is broken. It's worth rewriting from scratch just for this feature alone.

Posted by: aarone at January 13, 2005 11:52 AM

Easy solution? Leave in the existing per test setup and teardown, but add parallel one-time versions of these:

protected void setUpOnce() throws java.lang.Exception;
protected void tearDownOnce() throws java.lang.Exception

Posted by: j. betancourt at June 14, 2005 09:30 AM

I'm also annoyed about having instances for each test method, but one-time setup is trivial, no? There is still one test class, right?

public class TestMyClass {
static hasSetup = false;
public void setUp() {
if (hasSetUp) {
doMySetup();
hasSetUp = false;
}
}
}

Granted, I'd rather not do this.

Posted by: Michael Bushe at August 16, 2005 07:10 PM

Lots of discussion about JUnit and there have been some harsh edicts, like:

Cedric: "I am beginning to think we are once more stuck with a de facto standard that is utterly broken."

Utterly broken? I have produced enterprise applications for companies that have made millions of dollars and used JUnit for testing 80% of the code. Utterly is an extreme adjective here.

Cedric: "It creates unnecessary instances and can be very time consuming."

I guess I would like to see some metrics on this. Object creation, with todays hardward and OS's, is incredibly fast. I don't see what the problem is with unnecessary instances. For instance, I have ten unit tests that each have 3-6 test methods, and the suite runs in about 2 seconds. This is super fast. In my experience, the most JUnit tests I've had for a project was about 400, with probably (on average) 5 tests per class and it took about 15 minutes to run. And we once calculated that 5 minutes of that was dedicated to starting up and tearing down Hibernate.

Carlos: "then the setUp and tearDown mechanism sounds quite a bit redundant to me..."

In some cases that is true, but I write unit tests with the same design principles as my production code. So I use the setUp() method to prevent code duplication in the tests. By the way, I rarely override the tearDown() method. I try to write all tests such that they are responsible for setting up their but not tearing it down. That has another benefit too, if the test fails you can see the current state of the test. If tearDown() runs then your broken test data gets cleaned up.

Jason: "This notion that any method begining with 'test' in its name constituted a single unit test is excessively foreign to Java, despite the fact that numerous programmers seem to think it's a good idea to push things in that direction. The unit of design contract in Java is the Interface. Use it, instead of fighting to make the method name the unit of contract."

Well, a good argument in theory I suppose. I prefix all test methods with "test" because 1) it's easy to find the test and non-test methods in the class, and 2) it works well with the underlying JUnitRunner (it finds all test methods with the prefix "test"). I think pragmatically this argument is not that powerful, in reality the use of the prefix "test" is idempotent.

Also, comments about the Ant JUnit task are well founded, but I have managed to build successful software projects with this.

My last comment, I overall believe that JUnit works quite well with the Agile folks, because they (we) tend to write code in a way that makes it easy to test. So we don't often have to make massive fixtures and set up Tomcat and get JDBC Connections, put messages on queues. But JUnit isn't the write tool for every test scenario.

Posted by: Steven at August 17, 2005 11:34 AM

Just to add my 2 cents...
Steven says:

Cedric: "It creates unnecessary instances and
can be very time consuming."

Steven: I guess I would like to see some metrics on
this. Object creation, with todays hardward and
OS's, is incredibly fast. I don't see what the
problem is with unnecessary instances. For
instance, I have ten unit tests that each have
3-6 test methods, and the suite runs in about 2
seconds. This is super fast.

I am totally in agreement with Cedric. I am currently working on a project that uses WebWork+Spring+Hibernate and if I use the basic setUp and tearDown mechanism for setting up each test I get a test setUp time of about 5 seconds *for each* test. Multiply that by the number of tests and you get to what Cedric was saying about time consuming. The solution of sharing the initialized WebWork+Spring+Hibernate environment does reduce each test's execution time to a fraction of a second.

Reiterating what Stephane Bailliez said previously:
"in Ant 1.6.2 there has been a new forkmode attribute introduced. More on this in the documentation: http://ant.apache.org/manual/OptionalTasks/junit.html"

That solved it for me. Thanks Steve! :-)

Posted by: Francisco at June 8, 2006 12:36 PM

Just to add my 2 cents...
Steven says:

Cedric: "It creates unnecessary instances and
can be very time consuming."

Steven: I guess I would like to see some metrics on
this. Object creation, with todays hardward and
OS's, is incredibly fast. I don't see what the
problem is with unnecessary instances. For
instance, I have ten unit tests that each have
3-6 test methods, and the suite runs in about 2
seconds. This is super fast.

I am totally in agreement with Cedric. I am currently working on a project that uses WebWork+Spring+Hibernate and if I use the basic setUp and tearDown mechanism for setting up each test I get a test setUp time of about 5 seconds *for each* test. Multiply that by the number of tests and you get to what Cedric was saying about time consuming. The solution of sharing the initialized WebWork+Spring+Hibernate environment does reduce each test's execution time to a fraction of a second.

Reiterating what Stephane Bailliez said previously:
"in Ant 1.6.2 there has been a new forkmode attribute introduced. More on this in the documentation: http://ant.apache.org/manual/OptionalTasks/junit.html"

That solved it for me. Thanks Steve! :-)

Posted by: Francisco at June 8, 2006 12:37 PM

This works for me


private static int countTestCases = 0;

protected void setUp() throws Exception {


if(countTestCases == 0) {
log.info("oneTimeSetUp()");

// do some one time setup

countTestCases = this.countTestCases();
}
}

protected void tearDown() throws Exception {
--countTestCases;
if(countTestCases == 0) { // no more tests, tear down now
log.info("oneTimeTearDown()");

// do some one time tear down
}
}

Posted by: at September 6, 2006 02:05 AM

This works for me


private static int countTestCases = 0;

protected void setUp() throws Exception {


if(countTestCases == 0) {
log.info("oneTimeSetUp()");

// do some one time setup

countTestCases = this.countTestCases();
}
}

protected void tearDown() throws Exception {
--countTestCases;
if(countTestCases == 0) { // no more tests, tear down now
log.info("oneTimeTearDown()");

// do some one time tear down
}
}

Posted by: Ronny Nęss at September 6, 2006 02:06 AM

The first Commentor was correct. Just use junit.extensions.TestSetup. It wraps a Test and only calls setUp and tearDown once and only once, no matter how many testXXX() methods the wrapped Test has.

I found the best way to do this was to wrap the Test in a TestSuite it's Class ctor and in turn pass the TestSuite to the TestSetup:

public static Test suite()
{
return new MyTestSetup(new TestSuite(MyRealTest.class));
}

where MyTestSetup contains your "one-time methods" and MyRealTest contains the testXXX methods. Add more Classes to the throwaway TestSuite to have setUp and tearDown span across more than one Test.

NOW THE REAL BITCH IS ...

Ant doesn't let you automatically build up structures of TestSetups, TestDecorators, TestSuites, and Tests. will not save you. I'd rather never write a single suite() method and build it up in Ant instead. Something like:















testsetup@name and test@name will be FQCNs, but testsuite@name will be a string value identifier, not tied to a particular Java class -- more like an abstract container name.

And one should be able to nest TestSetups. So I can have "A" setup Hibernate and "B" setup JNDI with the deeper nested Tests having the benefit of both services being available.

Posted by: Toddius Zho at October 24, 2006 01:27 PM

nd one should be able to nest TestSetups. So I can have "A" setup Hibernate and "B" setup JNDI with the deeper nested Tests having the benefit of both services being available.

Posted by: roman at August 13, 2007 06:49 AM

And one should be able to nest TestSetups. So I can have "A" setup Hibernate and "B" setup JNDI with the deeper nested Tests having the benefit of both services being available.

Posted by: tramadol at September 24, 2007 05:46 AM

An excellent discussion on JUnit, I don't have anything to add as most of the senarios are discussed in detaild. But its a worth reading thread for any one who want to venture in to JUnit.

Posted by: Sekhar at October 5, 2007 01:12 PM

FYI The first two links of "all these deficiencies" don't link to any content.

Posted by: Merlyn Albery-Speyer at January 10, 2008 08:11 PM

If test cases shared the same fixture state and something failed during the tearDown of one test, the fixture would be left in an inconsistent state, and would cause confusing errors in subsequent tests running with the same fixture.

Posted by: artem at January 22, 2008 04:53 AM

I saw more of this staff at loadingvault.com

Posted by: cris at April 1, 2008 06:48 AM

Can some one tell me the purpose of
countTestCases = this.countTestCases();
in the code for escaping setUp() and tearDown() call during each testmethod's run?


Posted by: sriram at May 5, 2008 05:53 AM

I would like you to recomend filespump.com to look for files you are interested in.
Now it's the best file hosting search engine in the web.
More than 4 000 000 files indexed.
Try it and i think you will be satisfied with search results.
Use search string.

Posted by: Vlad at November 1, 2008 04:30 AM
Post a comment






Remember personal info?