Otaku, Cedric's weblog

I for one am happy to see some new ideas being explored for unit testing... However, I don't have time to add the stuff missing from TestNG that I need from JUnit and its surrounding tools (tool plugins, nice output, nice report generation, Clover integration, etc)... So, build it, and I will use it, but until then it's just nice ideas.

I really don't understand the arguments about problems with multiple frameworks. Unit testing is not rocket science, and just about any unit testing framework is not going to require any sort of massive investment to learn. I learned 95% of JUnit in a grand total of 15 minutes, mainly because it is so simplistic. A comprehensive unit testing framework might require a whole hour to learn :-)

I can understand worries about a multitude of frameworks if you're talking something complex - messaging, persistence, web stuff, etc. It can take weeks, months, even years to master frameworks of this time. Unit testing just does not fall into that category. If it takes you weeks or months to learn JUnit something is very, very wrong.

The implication in many web logs on JUnit I've seen is that they have some sort of intellectual investment in JUnit that they don't want to throw away. It's not said, but implied that they spent some time learning it and don't want to reinvest that time in something new. I say - how long did it take you to grok JUnit? For most uses you're dealing with TestCase, TestSuite, and Assert. What kind of "investment" was involved in learning these things?

From a pragmatic perspective, the sucky thing about moving away from JUnit is that of course people are extending concrete JUnit classes like TestCase and TestSuite. To get away from JUnit you'd have to re-write a ton and a half of unit testing code (particularly if you work for an XP company :-), or come up with a framework that hijacks the junit.framework package name space. Hmmm, I think I'll have to see what Cedric's compatability mode is all about :-)

For whatever its worth, here are my problems with JUnit. I've largely developed work-arounds.

1. Interface-based testing is awkward.
2. Only fast-fail tests are supported.
3. The design is brittle, poorly documented, and thus hard to extend.

How should interface-based testing be handled in TestNG? A nice feature would be bundled tests for interfaces in the java* namespace that are automatically applied. In other words:
- all classes will fail a (supressible) test if they violate the contract for equals() and hashCode()
- classes that implement Comparable will fail a test if they fail to implement Comparable
- and likewise for java.io.Serializable, java.util.Map, java.util.List, etc...

When I say all tests are fast-fail in JUnit, what I mean is that the first failed assertion in a method causes the test to exit. While this is usually desirable, a more conversational style of tests can sometimes be much easier to read and write. Such a conversational test doesn't generate a simple failure, but rather a score. The score could be either x of y passed or x percent passed. The important part is that the first failure doesn't terminate the test.

As I said, I've largely developed work-arounds for doing these in JUnit, but developing tools for conversational tests that play nice with the various JUnit runners was a real challenge. The exact contract that Eclipse expects of JUnit tests turns out to be different than what either of the bundled runners or the Ant task expect. Anyone who considers either the JUnit code or interfaces well-documented has a much different concept of well-documented than I do.

I liked your glib one liner about Struts.

However! I *DO* in fact, wish there were Servlets and only Servlets.*

And, as it turns out, I wish they'd never added regex**, and I think the new features being added in 1.5 are an abomination.***

*JSP and other similar technologies which work off the premise "oh, Java is too hard, lets come up with a 'dumbed down' version for 'the designers'" are so totally wrong. What they ignore is that for them to be 'useful' they're going to have to come up with something Turing complete, and also similar to Java - unfortunately they will fail in their stated goal, since any Turing Complete language is going to allow for the possibility of really ugly code, and then 'the designers' are going to have to become developers anyway, and the pre-existing developers are going to get tripped up by the subtle differences, and waste lots of time and get really frustrated.

**even the guy who wrote the Sun info page on the new regex features couldn't read his own regex several weeks later when I emailed him a bug fix for it!!!

***All sorts of subtle bugs are introduced, anything more than a really basic use of the features leads to the wheels falling off in ways that cannot be debugged (see also: 'reflection'). As a general rule, blackbox stuff should be behind the scenes, not in the limelight! Also anyone who doesn't understand casting is going to have bigger problems than just saving a couple of keystrokes!!

I don't care too much about multiple test frameworks, either way. I would, however, be pissed off if:

* I couldn't run _all_ my tests easily, with the click of one button (or one step in an Ant build, or whatever)

* In order to use the features of another test framework, I need to ditch my old one.

There are a few test frameworks out there (of which TestNG is nearly another contender, if you get that bug fixed. ;) that can run JUnit tests in addition to its own tests. Artima TestRunner is an example.

As far as web frameworks go: if I can use Struts and another web framework side by side, that's good. But I don't want to use a web framework which would prevent me using Struts on the same project; that would be bad.

Maybe we need a new JSR for defining unit test interfaces for test runner providers? :)

BTW, what happened to JUnit authors after they issued 3.8.1? Has anybody shot them down?