August 19, 2004Dependent test methodsImagine I am trying to test a server. In order to do this, my test class will contain the following test methods:
Obviously, the first two test methods listed above should be run before everything else. The way you would do this with JUnit is to put the first two methods in the initialization code, which will probably have to be static since JUnit instantiates a new object before each test method invocation, and we can't use setUp() either since it's invoked before each test method. This initialization code will set two booleans and the twenty test methods will have to test for these two booleans before proceeding, and if one of them is false, it will fail. Now let's assume that on a test run, I used the wrong JVM. In this case, JUnit will probably report this in the initialization code and it will then report twenty more failures for each test method. This is very bad for a variety of reasons:
This is why I believe that a test framework needs to provide support for "dependent test methods", where you can mark test method b() as depending on the successful run of test method a(). If a() failed, then b() will be marked as a SKIP, and not a FAIL. With such a feature, the test run will be marked "1 SUCCESS, 1 FAIL, 20 SKIPS", which is much more accurate. Here is how I would write this test using TestNG:
@Test
public correctVM() {}
@Test
public serverStartedOk() {}
@Test(dependsOnMethods = { "correctVM", "serverStartedOk" })
public method1() {}
...
This is okay but having to list all the methods that we depend for each new test method on is error-prone, so instead, let's use TestNG's groups:
@Test(groups = { "init" })
public correctVM() {}
@Test(groups = { "init" })
public serverStartedOk() {}
@Test(dependsOnGroups = { "init.* })
public method1() {}
...
We have gained a lot of flexibility with groups. For example, imagine that I want to add another init test method, such as "firewall is on". All I need to do is add this test method and declare that it is part of the group "init". Also, note that I used a regular expression in the "dependsOnGroups" declaration, as a reminder that you can actually define several init groups (such as "initOS", "initJVM", etc...) and they will automatically be run before any test method is invoked. But we can do even better. In the above example, I don't like the fact that whenever I add a new "real" test method, I need to remember to specify that it depends on the group "init.*". In TestNG, the traditional way to indicate that an annotation should apply to all test methods is to move this annotation at the class level. Also, I don't like the fact that "init" methods and "real" test methods are in the same class, so I'd like to use inheritance to provide a cleaner separation of roles. Therefore, I will restructure my tests like
this:
@Test(groups = "init")
public class BaseTest {
public correctVM() {}
public serverStartedOk() {}
}
This is now the base class for my tests. The @Test annotation is now on
the class, which means it applies to all the public methods inside that class
(so there is no need to repeat it on each individual method). Therefore,
each public method automatically becomes part of the group "init".Next, I write my test class as a subclass of BaseTest:
@Test(dependsOnGroups = { "init.*" })
public class TestServer extends BaseTest {
public method1() { ... }
public method2() { ... }
...
}
Here again, the @Test annotation is now on the class, which means that it
applies to all the public test methods, making it easier to add new testing
methods. Also, since this class extends BaseTest, it "sees" not only the
methods that are being inherited, but their annotations as well, so TestNG will
include all the methods from the base class that belong to the "init.*" group to
determine which methods need to be run first.
What's the overall result?
Posted by cedric at August 19, 2004 10:57 AM Comments
I have to say that this concrete example demonstrates the shortcomings of JUnit and the advantages of TestNG much more convincingly than anything else I have seen so far. Thank you very much, Cedric! Posted by: Christian Murphy at August 20, 2004 07:08 AMI don't buy your arguments. It just looks to me as though you're overlooking the "unit" aspect of JUnit. The premise of JUnit is that a TestCase is a collection of independant tests. The setUp method MUST therefore be called before each test because that is how the tests are made independant : its responsibility is to setup up the environement. TestNG is interesting but you keep bashing JUnit for things it was never designed for : non-regression testing (rather than plain, simple unit-testing) :) When did QA folks started using Junit for their testing ? ;-) Posted by: Deva at August 23, 2004 12:40 PMI have to agree that TestNG is looking more at end-to-end testing requirements than at programmer testing requirements. I think there is room for both tools (TestNG and JUnit) to have their use. I, for one, am involved in bringing JUnit out of stasis and breathing life into the project. I think that Cedric and I should work together, with Cedric capturing the testing market and JUnit retaining the programmer's market. I would love to see JUnit off the list of required skills for testers, replaced by TestNG. J. B. makes an interesting argument about the two different markets, but from my unlucky experience the best thing you can do for the QA folks is to give them a script that runs Ant (or whatever you use) which in turn invokes all required methods and they send you the results. I personally have never seen a QA person who had enough programmatic experience to do what is required to effectively run TestNG or JUnit tests by themselves, let alone know _what_ they are testing. I understand some of the JUnit purests point of view, but you look into any moderately complex application and you will probably not find 'pure' JUnit going on, but something more like what TestNG is actually intended to allow. I for one have run into this problem on my project, a framework, where you either have dependencies between tests or you fake this by giving a dependant test known good data, and therefore you can't really do a full throughput test, where you would like to take the results of test 'A' and if that test passes, and feed those results directly into test 'B', which may in turn cause test 'C' to run, etc. I have a ton of hard coded test information that is effectively configuration info, that I could more effectively do by having the tests that actually test the config files feed the tests that use the configuration. That would also get rid of any error prone programming that would come about because of all that hard coding. That is where the elegance of TestNG comes into play I believe. Posted by: Robert McIntosh at September 21, 2004 08:50 AMFundamentally junit performs individual tests as a single isolated unit with the setup and teardown functions providing the necessary isolation. The dependency aspect is not considered or atleast they are assumed correct. So I guess when Junit test are being performed, the dependency aspect could be safely ignored. However I see TestNG shine in the area of functional testing where it makes perfect sense to include such dependencies as mentioned. Many Thanks Cedric for the framework. Posted by: Simon John at December 6, 2004 08:30 PMPhilippe's point about overlooking the "unit" aspect of JUnit is semi-valid. That may well be what JUnit was initially intended for, but as time has passed it is now used as a general purpose testing framework. I for one have never used anything other than JUnit and its extensions for both unit and functional tests (except for UI tests). The problem is that the core of JUnit has not changed with the changes in it the way that developers use it. Posted by: Aramis at January 2, 2005 09:16 PMI get a cyclical error when tryint to implement the inheritance mechanism. Exception in thread "main" org.testng.TestNGException: I am working on TestNG for unit testing, first i installed jdk1.5 & in Myeclipse testNG was working fine. But am getting java.lang.UnsupportedClassVersionError: org/testng/remote/RemoteTestNG (Unsupported major.minor version 49.0) please let me know the reason asap........ Regards I am working on TestNG for unit testing, first i installed jdk1.5 & in Myeclipse testNG was working fine. But am getting java.lang.UnsupportedClassVersionError: org/testng/remote/RemoteTestNG (Unsupported major.minor version 49.0) please let me know the reason asap........ Regards How to create and run the TestNG script in intellij 6 IDEA. I also get the "Cyclic graph of methods" Exception from topologicalSort by following this example, using TestNG 5.5. However, it only seems to happen if the base class has the @Test(groups = "init") annotation on the class, not if it is on the method(s). Thus you may bypass the problem by having the annotation for the init group(s) on the method(s) of such a base class, extend this class, and then you may choose whether the the dependsOnGroup shall be on the classes or the methods - both work. Posted by: Endre Stølsvik at June 19, 2007 08:59 AMPost a comment
|