Otaku, Cedric's weblog

Announcing TestNG 5.0

I'm happy to announce the immediate availability of TestNG 5.0.

A lot has gone into this release, and instead of boring you with a standard product announcement, I thought I'd try something new by giving you not only the list of the new features (at least the most important ones) but also a quick explanation why this feature was added.  Read on, because this is no unusual product announcement!

Let's start with a quick overview of the main TestNG 5.0 features:

• The configuration annotations are now called @BeforeTest, @BeforeSuite, etc...
• The reports are now created in their own directory (one per suite) and a main index file shows a list of all the suites that were just run.  They have also received a series of improvements in appearance and content.
• It is now possible to specify the suite and test names directly in the @Test annotations, which allows you to do without a testng.xml.
• expectedExceptions has moved inside @Test.
• TestNG shows you friendly stack traces.
• Lightning Bolt:  Mana cost reduced approximately 17%.

Okay, I was just kidding about that last one.  Just making sure you're paying attention.

Let's go into details now.

Configuration annotations

In the early days of TestNG, the idea of an annotation that would indicate that a method has a special role was obvious to me.  What was not obvious was:  what should it be called?  Believe it or not, I was already hesitating back then between @BeforeSuite and @Configuration(beforeSuite = true).  That was two years ago.

I could find virtues in either of these approaches, but I eventually opted for the latter because of IDE support.  I thought that if all the configuration options can be gathered under one convenient annotation, it would be very easy for an IDE user to have a list of everything that's available:  they can just type @Configuration(<ctrl-space> and they would get the full list.

I still think this argument holds, but one can argue that it's equally easy to type @Before<ctrl-space> to get a list of all the "before types" that you can use.  On top of that, the newer style has two extra benefits:

• It's less verbose (compare the two versions above).
   
• It is more obvious that configuration annotations should not mix with each other.  That's a downside of the approach I initially chose:  while it is legal to specify several booleans (@Configuration(beforeSuite = true, beforeTest = true)), it doesn't make a lot of sense, and the shorter version makes this a bit clearer.

Of course, the old annotations are still recognized, but they are now deprecated.

New report structure

While TestNG lets you invoke multiple test suites easily (e.g. java org.testng.TestNG testng1.xml testng2.xml), the reports didn't maintain this distinction and bunched up all the results in the main output directory.  I thought I would take advantage of this release to clean up this structure and have each suite create its reports in its own directory (named after the suite name).  Additionally, the output directory now contains a single index.html that gives a composite view (and links) of all the suites that were run along with their outcome.

Also, the reports are now using a CSS file and their appearance has been improved.  They also contain additional information, such as a pointer to the testng.xml that was used for a particular suite.

suiteName and testName

These two attributes belong to the @Test annotation and they can be used as follows:

@Test(suiteName = "My suite", testName = "My test")
public class Test {

This feature will be of interest to users who don't want to use a testng.xml and, instead, prefer to use the command line or ant to invoke their test suite.  With these two new attributes, these users can now get the benefit of seeing their test suites end up in a specific report.

Of course, classes that use these attributes will have their reports generated in the structures as described in the previous paragraph.

expectedExceptions

This attribute used to have its own annotation (@ExpectedExceptions), but it was pointed to me that this annotation made little sense if it wasn't coupled with a @Test annotation, so this was definitely a design mistake that needed to be corrected.  As of 5.0, @ExpectedExceptions is deprecated and you should now use the new expectedExceptions attribute from @Test:

@Test(expectedExceptions = NumberFormatException.class)
public void shouldThrow() {

Friendly stack traces

This was another heavily-requested feature.  Whenever a test fails, it does so with an AssertionException which shows the entire stack trace, including TestNG's internal frames, reflection, etc...  These stack traces will now be hidden by default so that the report only shows what you really care about:

Before:

java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:396)
at java.util.Properties.setProperty(Properties.java:128)
at org.testng.xml.XmlSuite.toXml(XmlSuite.java:318)
at org.testng.reporters.SuiteHTMLReporter.generateXmlFile(SuiteHTMLReporter.java:81)
at org.testng.reporters.SuiteHTMLReporter.generateReport(SuiteHTMLReporter.java:72)
at org.testng.TestNG.run(TestNG.java:613)
at test.configuration.BeforeTestOrderingTest.verifyBeforeTestOrdering(BeforeTestOrderingTest.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.testng.internal.MethodHelper.invokeMethod(MethodHelper.java:552)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:411)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:785)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:110)
at org.testng.TestRunner.privateRun(TestRunner.java:687)
at org.testng.TestRunner.run(TestRunner.java:567)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:242)
at org.testng.SuiteRunner.run(SuiteRunner.java:148)
at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:899)
at org.testng.TestNG.runSuitesLocally(TestNG.java:861)
at org.testng.TestNG.run(TestNG.java:611)
at org.testng.TestNG.privateMain(TestNG.java:999)
at org.testng.TestNG.main(TestNG.java:936)

After:

java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:396)
at java.util.Properties.setProperty(Properties.java:128)
at test.configuration.BeforeTestOrderingTest.verifyBeforeTestOrdering(BeforeTestOrderingTest.java:47)
... Removed 21 stack frames, click here to show them.

Finally, here is the complete TestNG 5.0 change log:

• Added: Stack traces can be interactively shown in the HTML reports
• Added: Link to testng.xml in the reports
• Added: Ant task: support for JVM, workingDir, timeout
• Added: New structure for reports, suites go in their individual directory
• Added: @Test(suiteName) and @Test(testName)
• Added: The stack traces in reports do not include TestNG frames (system property testng.exception)
  see: http://groups.google.com/group/testng-dev/browse_thread/thread/9f4d46ade10b0fda)
• Fixed: Exit with error when no methods are run
  (see: http://groups.google.com/group/testng-dev/browse_thread/thread/3c26e8a5658f22ac)
• Added: List of methods in alphabetical order
• Fixed: Class-scoped annotations were not recognized when inherited
• Added: Deprecated @Configuration and introduced @BeforeSuite/Test/Class/TestMethod
• Added: Deprecated @ExpectedExceptions and moved it into @Test
• Added: expectedExceptions to @Test, deprecated @ExpectedExceptions
• Added: New annotations: @BeforeSuite, @BeforeTest, etc...
• Fixed: Was returning an exit code of 0 if a cyclic graph was detected
• Added: Interface org.testng.ITest so that tests can declare a name
  Fixed: The Text reporter was reporting the square of the actual number of methods
• Fixed: Bug reported by Eran about dependencies with an afterClass method
• Added: IHookCallBack now receives the ITestResult in its run() method
• Added: Name of suite for command line can be set with -Dtestng.suite.name=xxx
• Fixed: TestNGAntTask was hardcoding m_haltOnFSP to true
• Fixed: Passing a null parameter caused an NPE in the reports
• Added: "listener" to the ant task (and documentation)
• Added: if patch-testng-sourcedir.properties is found in the classpath
  with a property "sourcedir" containing a ; separated list of
  directories, this list will override -sourcedir.

Download TestNG 5.0.

Posted by cedric at July 19, 2006 09:06 AM