Otaku, Cedric's weblog

Announcing TestNG 5.1: making testing easier, one thread at a time

TestNG 5.1 is now available for download from http://testng.org.  The change log is included below and contains a lot of bug fixes, but also a particular feature I'd like to expand on.

TestNG makes it very easy for you to run your tests in separate threads, which provides a very significant speed up in a lot of cases (this particular user said that switching to multithreaded tests reduced their test run times from forty minutes down to... four minutes!).  However, when you do this, you can encounter various problems if the classes you are testing are not multithread-safe (which is very often the case).

For example:

public class ATest {
  private A a;

  @BeforeClass
   public void init() {
    this.a = new A();
  }

  @Test
  public void testf1() {
    // test a.f1();
  }

  @Test
  public void testf2() {
    // test a.f2();
  }
}

In this example, the two test methods testf1() and testf2() test the methods A#f1 and A#f2 respectively, but when you ask TestNG to run these tests in parallel mode, these two methods will be invoked from different threads, and if they don't properly synchronize with each other, you will most likely end up in a corrupt state.

One way to solve this problem is to declare that the test methods depend on each other, thus forcing TestNG to run them in the same thread, but that's obviously tedious and also not semantically correct (they don't really depend on each other).

Another possibility is to make A#f1 and A#f2 properly synchronized, but again, it's a lot of work just to enable testing, and while I'm fine with making minor modifications to my classes to make them easier to test (making certain methods more visible, adding accessors, etc...), I think that making my classes multithread-safe crosses the line.

Therefore, TestNG 5.1 provides a new attribute in the @Test annotation, which can be specified at the class level:

@Test(sequential = true)
public class ATest {
  ...

When TestNG sees this flag, it will make sure that all the test methods in the given class are always run sequentially (right now, it uses the simplest way to achieve this goal:  run all the test methods in the same thread, but there are other ways to do this).

An interesting consequence of this fix (which was trivial to make since support for sequential runs this was already in TestNG because of dependencies) is that the workers used by TestNG can now contain either:

• A single test method.
• A set of ordered methods (if dependsOnGroups or dependsOnMethods is being used).
• A set of unordered methods (if @Test(sequential = true) was used).

Obviously, the last two workers are going to be holding on to threads longer than the first one, but I still think that the runs will be faster than if the parallel flag is not set.

Note that the link to the thread of discussion that I posted above (here it is again) is a good example of how new features are added to TestNG:

• A user posts a request for a feature.
• I ask for other people to comment on the issue, we validate the need.
• Once the feature has been deemed useful, a few emails are exchanged on the proper way to add it to TestNG.
• The feature is implemented, tested and I upload a beta on http://testng.org for immediate validation by the user.

Here is the complete change log for TestNG 5.1 (big thanks to Alexandru for all the bug fixing!)

Enjoy!

Core

• Added: @Test(sequential = true)
• Fixed: TESTNG-102 (Incorrect ordering of @BeforeMethod calls when a dependency is specified)
• Fixed: TESTNG-101 (HTML output contains nested <P> tags and a missing <tr> tag)
• Added: support for specifying test-only classpath (http://forums.opensymphony.com/thread.jspa?mesageID=78048&tstart=0)
• Fixed: TESTNG-93 (method selectors filtering @BeforeMethod)
• Fixed: TESTNG-81 (Assert.assertFalse() displays wrong expected, actual value)
• Fixed: TESTNG-59 (multiple method selectors usage results in no tests run)
• Fixed: TESTNG-56 (invocation of @Before/AfterClass methods in parallel/sequential scenarios)
• Fixed: TESTNG-40 (failures suite does not contain @Before/After Suite/Test methods)
• Fixed: TESTNG-37 (allow passing null parameter value from testng.xml)
• Fixed: TESTNG-7 (display classname when hovering method)

Eclipse plug-in

• Added: run contextual test classes with parameters from suite definition files
• Added: TESTNG-100 (Show HTML reports after running tests)
• Added: TESTNG-97 (Double click top stack to raise comparison)
• Added: TESTNG-84 (plug-in UI for suite option should support absolute path)
• Added: TESTNG-20 (copy stack trace)
• Fixed: TESTNG-72 (display groups with non-array values)
• Fixed: TESTNG-64 (Eclipse plug-in applies added groups to all launch configurations)
• Fixed: TESTNG-28 (Cannot select groups from dependent eclipse projects)
• Fixed: TESTNG-25 (do not display fully qualified method name when running contextual test class)

Improved behavior

• TESTNG-98 (temporary files have guaranteed fixed names)
• TESTNG-95 (Assertion failed comparison trims trailing ">")
• TESTNG-70 (TestNG prevents eclipse from opening an older CVS version of a java class)
• Display of test hierarchy information (TESTNG-29)

Posted by cedric at August 18, 2006 09:38 AM