April 28, 2004

TestNG: Testing, the Next Generation

It's been a few weeks since I announced another killer app, hasn't it?

I have complained about JUnit many times, both in terms of design and implementation.  I think my complaints are justified, but if there is something I don't like, it's people who complain but don't do anything to solve the problem they are complaining about.

Therefore, without further ado, I give you...  TestNG, the JUnit killer.

For quite a while, I have seen very little reason to write a replacement to JUnit since I didn't think the added value would be enough to convince the vast number of JUnit users to switch.  After reading up about NUnit, it occurred to me that using JSR 175 annotations for testing was an interesting evolution, but still not quite the quantum leap I expected (besides, in its effort to mimick JUnit, NUnit repeats a lot of its flaws).

So I went back to the drawing board, in search of one last killer argument to justify taking on this endeavor, and I came up with another ultimate complaint that, strangely, I hadn't voiced about JUnit:  the test Suite.

I have always been confused by JUnit's Test Suites.  Where should the method belong?  In the test class?  Separately?  Should it be static?  And if I want to change the tests run, why do I need to rebuild my code?

With these changes in mind, I came up with the design principles that are the foundation of TestNG.  To quote the documentation:

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:

  • JSR 175 Annotations.
  • Flexible test configuration.
  • Default JDK functions for runtime and logging (no dependencies).
  • Powerful execution model (no more TestSuite).

I think the combined use of JSR 175 annotations and Test Groups make TestNG a very flexi...  (augh, sorry Hani)...  a very supple testing framework that should compare decently with JUnit.

Check out the documentation, take it for a run and let me know what you think.

Posted by cedric at April 28, 2004 02:10 AM

What about apricot? http://apricot.sourceforge.net/

It's no not flexible but it is simple, open and extensible apparently.

Posted by: Glen Stampoultzis at April 27, 2004 08:42 PM

hmmm, the apricot smells a bit rotten. Last release, 0.1.4, was made in March 2002.

Posted by: Brendan Humphreys at April 27, 2004 09:32 PM

It is not clear why JUnit TestSuite and the group concept are different -- for example, why not use annotations to generate a TestSuite based on "group
" tags ...

Posted by: Chandrashekara K A at April 27, 2004 10:19 PM

I'll check this out and take it for a spin, but...

I'm not sure how this buys me anything over the reflection-based support in JUnit. A key thing to remember is that the reflection support is there as a "best-effort" approach; it saves having to roll test suites by hand. If you want to take a different approach, it's trivial to do (including the TestCase that creates only one instance).

If the annotation method doesn't retain this flexibility, there would be problems. To be honest, I'd like a way I can blend the techniques.

A number of the other problems that you metioned are with different tools (Ant, DbUnit), and not really limitations of JUnit itself.

Oh, and if you want _me_ to adopt this (ah, vanity!) you'll need to give the XML and XSLT support that come in various JUnit-aware tools (such as the JUnitReport task in Ant, and CruiseControl).

Posted by: Robert Watkins at April 27, 2004 10:45 PM

Interesting. NUnit is definitely a much nicer experience than JUnit (and of course, it's written by ThoughtWorkers ;-) ).

There's two things though:
1. IDE integration. The JUnit test-runner in IntelliJ (and to a lesser extent the one in Eclipse) is just superb. Can you make a JUnit compatability layer?
2. XML log format. There's a massive amount of tools for working with logs in the format produced by the ant-task, there's CruiseControl and JUnitReport, but I have also written billions of small nifty scripts that does analysis, graphing, reports and so on.

If you want to compete, you have to address these issues in some way.

Posted by: Jon Tirsen at April 28, 2004 02:09 AM

Sometimes it's best just to put your hand in your pocket and buy some decent software, XDE Tester.

Posted by: Geoff at April 28, 2004 07:47 AM

Check out http://www.artima.com/suiterunner/, Bill Venners' attempt to kill JUnit. Didn't happen. JUnit is entrenched, man. At the very least, you probably need to provide a bridge for all the JUnit tests out there. Your thing has to be able to run JUnit tests for it to be adopted, I'd venture.

I like the idea of using attributes though. But I guess I feel a bit of "If he spent more time writing tests and using JUnit, he'd probably have less problems with it." I've used it a lot over the years (and contributed chapters to a book about it -- forthcoming from Manning, yada yada). Most of the problems people have are with the test runners in the default junit.jar, but most people who use JUnit in real projects *don't use* those test runners. Everyone uses Ant or an IDE-based test runner that has none of those problems. I haven't written a TestSuite in years! I don't need to for anything whatsoever because Eclipse and Ant provide ways to easily run tests individually or in batches without relying on TestSuites.

Posted by: Scott Stirling at April 28, 2004 10:54 AM

I read your announcement of TestNG and found it very interesting. Here's what I'd like to tell you from my enjoying it.

In order to build and run TestNG, you need to set JAVA_HOME=tiger and CLASSPATH=testng.jar. Without the settings, I couldn't see it working. I think it will be good for http://beust.com/testng/ documentation to explain them.

2. Configuration annotation
Configuration seems like a Marker, so my suggestion is declaring

public @interface BeforeTestClass {}
public @interface AfterTestClass {}
public @interface BeforeTestMethod {}
public @interface AfterTestMethod {}

and using them like

public class BaseTest {
public void setUp() {

public void tearDown() {

public void calledBeforeEachTest() {

public void calledAfterEachTest() {

for more brevity and readability.

Posted by: Ias at April 28, 2004 09:25 PM

This looks like cool stuff. You may find my XSTest library interesting. It takes functional test scripts written in Jython and turns them into scalability/load tests. XSTest is at http://www.pushtotest.com/Docs/howto/xstest.html.


Posted by: Frank Cohen at May 1, 2004 12:26 PM

It's interesting, but so far it just looks like another way to implement JUnit, rather than a replacement for it. It would probably help me if you would describe your specific motivation for certain, specific features in TestNG. As an example, what's the problem with extending TestCase? I haven't found the NUnit-style test case marking to be such a great improvement over the way I do things with JUnit.

I'm trying not to be biased towards JUnit -- admittedly, that's tough for me to do -- but it seems strange even to look at something new without understanding the motivation for it. You say that TestSuite confuses you, but it doesn't confuse me, so I just don't know what the big deal is.

I must say that I like the attribute-based way to creating several, overlapping test suites (groups). That's nice, but we could easily do that on top of JUnit; no need to start from scratch.

I'll keep an eye on it, and I /am/ interested, but I just don't see the point yet.

Posted by: J. B. Rainsberger at May 4, 2004 02:08 PM
Post a comment

Remember personal info?