Posts Tagged testng

TestNG anniversary

I was updating the TestNG home page to include the information about the recent availability of version 5.12 in the Maven repository (check it out!) when I noticed that I created this page exactly six years and one day ago.

Wow… Six years.

I believe it took me a few months to implement v1.0, so TestNG is actually a bit older than that.

If you had asked me then if I thought I would still be working on TestNG six years later, I would have politely nodded while wondering from what mental institution you escaped.

Big thanks to the TestNG community for accompanying me through this fantastic journey, and looking forward to more awesome testing, next generation style.

Tags:

Better mock testing with TestNG

When you are using mocks in your tests, you usually follow the same pattern:

  1. Create your mock.
  2. Set it up.
  3. Play it.
  4. Verify it.

Of these four steps, only steps 2 and 3 are specific to your tests: creating and verifying your mock is usually the exact same code, which you should therefore try to factor out. It’s easy to create your mock in a @BeforeMethod but the verification aspect has never been well integrated in testing frameworks.

Until today.

Mock verification, or more generally, systematic invocation of certain test methods, is a feature that’s been requested several times by TestNG users and while I’ve often been tempted to go ahead and implement it, something has always felt “not right” about it.

As it turns out, there are apparently three ways that you can automate mock verification in TestNG: one that is incorrect, one that doesn’t work and one that actually works.

Let’s start with the incorrect one:

  @BeforeMethod
  public void init() {
    this.mock = // create mock
  }

  @Test
  public void t1() {
    // set up and play mock
  }

  @Test
  public void t2() {
    // set up and play mock
  }

  @AfterMethod
  public void verify() {
    this.mock.verify();
  }

This approach will run the verification after each test method, but the problem is that this code is being run in a configuration method (@AfterMethod) instead of a test method (@Test). Configuration methods are handled differently by TestNG in how they fail and how they get reported (in a nutshell, a configuration method that fails typically aborts the entire run since your test environment is no longer stable).

My next thought was to use TestNG’s groups and dependencies:

  @BeforeMethod
  public void init() {
    this.mock = // create mock
  }

  @Test(groups = "mock")
  public void t1() {
    // set up and play mock
  }

  @Test(groups = "mock")
  public void t2() {
    // set up and play mock
  }

  @Test(dependsOnGroups = "mock")
  public void verify() {
    this.mock.verify();
  }

At least, this solution runs the verification in a @Test annotation, but because of the way dependencies are run, the order of invocation for the code above will be t1(), t2() and then verify(). This is obviously wrong since we will only be verifyin whatever mock was played last. You could consider having each test method create and store a different mock and then have verify() go through all these mocks and calling verify() on them, but it’s clearly a subpar solution.

So let’s turn our attention to the correct solution, which involves the use of a method interceptor.

I already feature method interceptors in a previous entry that showed you how to create a new annotation @Priority that allows you to order your methods. The idea is similar here, except that instead of reordering the methods that we want TestNG to run, we are going to change their number as well.

For this to work, we introduce two new annotations: @Verify which indicates that this method needs to be verified, and @Verifier, which annotates the method that performs the verification. Here is an example of how you use them:

  @Verify
  @Test
  public void t1() {
    // set up and play mock
  }

  @Verify
  @Test
  public void t2() {
    // set up and play mock
  }

  @Verifier
  @Test
  public void verify() {
    this.mock.verify();
  }

Now we need to write a method interceptor that will go through all the @Test methods, locate the verifier and then return a list of methods where each method annotated with @Verify is followed by the invocation of the @Verify method. The code is straightforward:

IMethodInterceptor mi = new IMethodInterceptor() {

  public List<IMethodInstance> intercept(List<IMethodInstance> methods,
      ITestContext context) {
    List<IMethodInstance> result = Lists.newArrayList();
    IMethodInstance verifier = null;

    // Doing a naive approach here: we run through the list of methods
    // twice, once to find the verifier and once more to actually create
    // the result. Obviously, this can be done with just one loop
    for (IMethodInstance m : methods) {
      if (m.getMethod().getMethod().getAnnotation(Verifier.class) != null) {
        verifier = m;
        break;
      }
    }

    // Create the result with each @Verify method followed by a call
    // to the @Verifier method
    for (IMethodInstance m : methods) {
      if (m != verifier) {
        result.add(m);
      }

      if (m.getMethod().getMethod().getAnnotation(Verify.class) != null) {
        result.add(verifier);
      }
    }

    return result;
  }

};

Running it produces the following output:

t1
Verifying
t2
Verifying

With this simple interceptor (full source), you can now completely factor out the boiler plate logic of your mocks and focus on their business logic.

Happy mocking!

Tags:

More on multithreaded topological sorting

I received a few interesting comments on my previous entry regarding the new multithreaded topological sort I implemented in TestNG and there is one in particular from Rafael Naufal that I wanted to address:

The @Priority annotation couldn’t be adapted to say which free methods get scheduled first? BTW, the responsibility of knowing which nodes are free couldn’t be moved to the graph of test methods?

This is pushing my current algorithm even further in the sense that we not only want to schedule free nodes as they become available, we also want to schedule them in order of importance.

What makes a node more important than another? Its level of dependencies. The more a node is depended upon, the more beneficial it is to schedule it as soon as possible since this will end up freeing more nodes. Admittedly, you are still bounded by the size of your thread pool, but this is exactly what we want: increasing the pool size should lead to more parallelism and therefore better performance, but the current scheduling algorithm being fair (or random) means that we are not guaranteed to see this performance increase.

My first reaction was to modify my Executor but as it turns out, you can actually do this with the existing implementation. The constructor of all the Executors takes a
BlockingQueue in parameter, which is the queue that the Executor will use to process the workers. Unsurprisingly, there already is a priority queue called PriorityBlockingQueue.

All you need to do is to use that queue instead of the default one when you create your Executor and then make sure that the workers you pass it have a natural ordering. In this case, the weight of a worker is how many other workers depend on it, which is very easy to calculate.

On a related topic, I wanted to get a closer look at how the algorithm I described in my previous blog post actually works. I described the theory and I have tests that show that it seems to work as expected, but it occurred to me that I could actually “view it from the inside” with little effort.

First, I added a method called toDot which generates a Graphviz file representing the current graph. This turned out to be trivial:

/**
* @return a .dot file (GraphViz) version of this graph.
*/
public String toDot() {
  String FREE = "[style=filled color=yellow]";
  String RUNNING = "[style=filled color=green]";
  String FINISHED = "[style=filled color=grey]";
  StringBuilder result = new StringBuilder("digraph g {\n");
  Set<T> freeNodes = getFreeNodes();
  String color;
  for (T n : m_nodesReady) {
    color = freeNodes.contains(n) ? FREE : "";
    result.append("  " + getName(n) + color + "\n");
  }
  for (T n : m_nodesRunning) {
    color = freeNodes.contains(n) ? FREE : RUNNING;
    result.append("  " + getName(n) + color + "\n");
  }
  for (T n : m_nodesFinished) {
    result.append("  " + getName(n) + FINISHED+ "\n");
  }
  result.append("\n");
  for (T k : m_dependingOn.getKeys()) {
    List<T> nodes = m_dependingOn.get(k);
    for (T n : nodes) {
      String dotted = m_nodesFinished.contains(k) ? "style=dotted" : "";
      result.append("  " + getName(k) + " -> " + getName(n) + " [dir=back " + dotted + "]\n");
    }
  }
  result.append("}\n");
  return result.toString();
}

Then I modified the executor to dump the graph every time a worker terminates, and finally, I wrote a shell script to convert these dot files into images and to create an HTML file. I ran a simple test case, processed the files with the shell script and here is the final result.

A yellow node is “free”, green means that the node is “ready” (to be run in the thread pool), grey is “finished” and white nodes haven’t been processed yet. Dotted arrows represent dependencies that have been satisfied.

As you can see, the execution matches very closely what you would expect based on my description of the algorithm and I confirmed that changing the size of the thread pool creates different executions.

Tags:

Test method priorities in TestNG

A TestNG user recently requested an interesting feature: method priorities.

The idea is to be able to assign a priority to a test method (or an entire class) to guarantee that it will be run before other methods with a higher priority. Note that this is slightly different from dependent methods, since it’s just about ordering: if a method with a priority 2 fails, methods with higher priorities will still be run.

As it turns out, this is very easy to implement with one of TestNG 5.8 newest features: method interceptors.

TestNG lets you declare a class implementing the interface IMethodInterceptor, whose signature is:

public interface IMethodInterceptor extends ITestNGListener {
  List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context);
}

TestNG passes the list of methods it’s about to run so you get a chance to reorder them. Implementing method priorities with this interface is very simple.

Let’s start with what we want this feature to look like. In the following example, I declare three kinds of priorities:

  • Priorities at the method level.
  • A priority at the class level.
  • No priority. When no priority is defined on a method, we want to look for a priority on the class first, and if none can be found, the priority will receive a default value of 0.

Here is our test:

@Priority(10)
public class PriorityTest {
  @Test
  public void b1() { System.out.println("Default priority");}

  @Priority(-3)
  @Test
  public void a1()  { System.out.println("Priority -3");}

  @Priority(-3)
  @Test
  public void a2()  { System.out.println("Priority -3");}

  @Priority(3)
  @Test
  public void c1()  { System.out.println("Priority 3");}

  @Priority(3)
  @Test
  public void c2()  { System.out.println("Priority 3");}

  @Priority(4)
  @Test
  public void d1()  { System.out.println("Priority 4");}
}

Next, we implement the @Priority annotation:

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface Priority {
  int value() default 0;
}

Fairly straightforward: we make the annotation valid on methods and types only, and we use the shortcut value() annotation so we can specify @Priority(1).

Now comes the implementation of the IMethodInterceptor:

public class PriorityInterceptor implements IMethodInterceptor {
  public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
    Comparator<IMethodInstance> comparator = new Comparator<IMethodInstance>() {
      private int getPriority(IMethodInstance mi) {
      int result = 0;
      Method method = mi.getMethod().getMethod();
      Priority a1 = method.getAnnotation(Priority.class);
      if (a1 != null) {
        result = a1.value();
      } else {
        Class cls = method.getDeclaringClass();
        Priority classPriority = cls.getAnnotation(Priority.class);
        if (classPriority != null) {
          result = classPriority.value();
        }
      }
      return result;
    }

    public int compare(IMethodInstance m1, IMethodInstance m2) {
      return getPriority(m1) - getPriority(m2);
    }
  };

  IMethodInstance[] array = methods.toArray(new IMethodInstance[methods.size()]);
  Arrays.sort(array, comparator);
  return Arrays.asList(array);
}

The heart of this implementation is our Comparator, which tries to find a @Priority annotation the method it receives in parameter. If it can’t find any, it looks on the declaring class, and failing to find one there as well, assigns it a default value. The intercept() method simply sorts the methods with this comparator and returns it.

Now, let’s run our test without using that interceptor:

<suite name="Example">
  <test name="Method interceptor example" >
    <classes>
      <class name="priority.PriorityTest" />
    </classes>
  </test>
</suite>

As expected, the output is fairly random:

Priority 3
Default priority
Priority 3
Priority -3
Priority -3
Priority 4

Now we declare our interceptor in the testng.xml file:

<suite name="Example">
  <listeners>
    <listener class-name="priority.PriorityInterceptor" />
  </listeners>

  <test name="Method interceptor example" >
    <classes>
      <class name="priority.PriorityTest" />
    </classes>
  </test>
</suite>

And we get the expected outcome, with our method without any @Priority receiving the class priority (10) since it didn’t define any itself:

Priority -3
Priority -3
Priority 3
Priority 3
Priority 4
Default priority

To verify that our class priority implementation works, we can remove the @Priority class annotation. In this case, our method without any @Priority should receive a priority of 0:

Priority -3
Priority -3
Default priority
Priority 3
Priority 3
Priority 4

As you can see, Method Interceptors allow you to replace TestNG’s ordering of test methods with your own. It’s particularly powerful if used in conjunction with annotations, as shown in this example, but it’s also possible to imagine other external configuration factors to customize TestNG’s running engine to your liking.

Tags: