Archive for September, 2004

AOP still not there

I wish Dion were right when he says that AO is gaining use, or at least interest. Indeed, we have seen a significant pick up in interest in AOP these past two years, but unfortunately, not much progress has been made since then in terms of visibility.

The sad truth is that the only times AOP makes an appearance is in bleeding edge conferences such as No Fluff Just Stuff or The ServerSide Symposium. Are you seeing it around you in your every day job? I certainly don’t, and actually, most of the developers I talk to don’t even know what it is.

What went wrong? Failure to communicate or simple inadequacy to the problems software developers are facing every day?

Is AOP doomed to be a niche technology reserved to a tiny fraction of the developers?

Better reporting in TestNG

I have improved the reporting made by TestNG to give you more information on your last test run, and more precisely:

    The list of groups with their methods. When you write your test methods and assign them to certain groups, you might lose track of which method belongs to which groups since these groups can span over different classes. This view allows you to quickly see the list of groups and which methods are in them, which makes it easy to see, for example, which tests are in the “broken” group (and therefore are in need to be fixed) or how many methods your “functest” group really contains.
    The sequence of method invocations. This is a list of all the methods that were invoked during your test, sorted chronologically. This is a good way to make sure that your configuration methods are being invoked correctly or that your inclusion and exclusion of test methods is as you expect it. As you can see from this view, configuration class methods are invoked before the instantiation of your test class and after the last test method has run, while configuration methods methods are being invoked “around” each test method.

I am still not quite satisfied with the way the main page looks, so if you have suggestions to improve it, please let me know.

suite() : good or bad?

I have a love-hate relationship with JUnit’s suite() feature.

Quick reminder: a JUnit class can either have test methods (starting with test) or a suite() method such as this:

public static Test suite() {
TestSuite suite= new TestSuite();
suite.addTest(new MoneyTest("testMoneyEquals"));
suite.addTest(new MoneyTest("testSimpleAdd"));
return suite;
}

In which case, the Tests returned by your suite() method will be inspected recursively for either test methods or more suites.

This is a powerful concept that allows you to define an entire hierarchy of tests easily. Typically, you can end up with a “root” suite such as:

public static Test suite() {
TestSuite suite= new TestSuite();
suite.addTest(new BackEndSuite());
suite.addTest(new FrontEndSuite());
return suite;
}

and in turn, have BackEndSuite() contain more suites (DataBaseSuite, SecuritySuite, UserProfileSuite, etc…) and FrontEndSuite() contain things such as HTMLSuite, TemplateSuite, PostSuite, etc…

The problem comes from debugging a failed test. Suddenly, it becomes really hard to pinpoint exactly what test is failing because, well, you simply can’t tell right away which tests are active, enabled, and what suite they belong to.

In short, there is not a single file that you can look at and get an overall view of your test structure.

Confronted with this problem with TestNG, I am not sure what to do. I like the idea of the hierarchy but I wonder if it doesn’t have more downsides than upsides.

For now, I have decided to try the “one file only” approach, and that file is your build.xml. The ant task allows you to specify a fileset, so you would have something like this:

<testng outputDir = "${basedir}/test-output">
<fileset dir="${basedir}">
<include name="back-end/database/testng.xml" />
<include name="back-end/security/testng.xml" />
<include name="front-end/html/testng.xml" />
...
</fileset>
</testng>

If I wanted to introduce the concept of a hierarchy of tests, I would simply allow testng.xml to reference other testng.xml files.

Thoughts?

Announcing TestNG 1.0!

I am happy to announce the availability of TestNG 1.0. A lot of changes have been made since the early releases thanks to the recent addition of Jolly Chen to the team.

As a quick reminder, TestNG is a testing framework using annotations to provide a more powerful way to test your code in various ways: unit, regression, functional, integration, etc… TestNG allows you to clearly separate your Java code from the way your tests are run. You never need to recompile any of your classes if you decide to run a different set of tests or suites.

TestNG also provides a very flexible configuration mechanism allowing you to specify arbitrary methods to be invoked at particular moments during your test run, such as “before every test method” or “after all the test methods have run”.

Here is a quick example of what TestNG code looks like:

import com.beust.testng.annotations.*;
public class SimpleTest {
@Configuration(beforeTestClass = true)
public void setUp() {
// code that will be invoked when this test is instantiated
}
@Test(groups = { "functest" })
public void itWorks() {
// This method will be invoked if the current run
// includes the group "functest"
}
}

Some of the major changes since the previous versions include:

  • XML. The property files were easy to work with in the beginning but it was quickly obvious that they wouldn’t scale. Now, the information for a suite run (which contains several tests, each test containing several classes) is conveniently located in one XML file.
  • Dependent methods. With this feature, you can guarantee that certain groups of methods are always invoked before others. If the dependents didn’t run or fail, the method will be marked as a SKIP, which can be seen as less critical than a FAILURE since SKIPs will typically be automatically resolved when the FAILURE that caused them is fixed.
  • Groups of groups. Once you have specified that your test methods belong in specific groups, you can specify bigger groups, giving you a lot of flexibility on exactly what tests are run each time.
  • Better reporting. Not only does TestNG have its own HTML reporting, it is also compatible with JUnitReport. You can see an example here.
  • Open API. TestNG exposes its engine with a convenient set of listeners which allow you to plug-in your own reporting mechanism or for any other purpose (as a matter of fact, the JUnitReport code is such a listener). It is also trivial to invoke TestNG programmatically, thus skipping the XML phase and providing TestNG with direct Java objects.

The XML is fully validate and has a public DTD, but here is a quick description. testng.xml lets you specify a test Suite, which contains parameters and tests. Tests contain three kinds of data:

  • Parameters, which override the Suite parameters.
  • Groups, made of two parts, definitions and runs.
  • Classes, which define which classes are going to be part of this test run

In turn, groups are made of two parts:

  • Definitions, when you want to create groups that contain other groups.
  • Runs, which specify the groups to be run.

The TODO list for future versions is pretty long but this version should give you a good idea of the future directions.

Download TestNG and try it, you won’t regret it!