Otaku, Cedric's weblog

A name like assertTrueButContinue might be more descriptive.

I also dislike having to manage StringBuilders separately. A chained way assertTrueButContinue( selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled").assertTrueButContinue(selenium.isElementPresent("id=txtfiled"), "Couldn't find submit").summarize(); may be more consistent.

interface ComplexAssert {
  ComplexAssert add(boolean success, String message);
  void requireAll();
  void requireAny();
}
ComplexAssert ca = ...;
ca.add(selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled")
  .add(selenium.isElementPresent("id=txtfiled"), "Couldn't find submit")
  .requireAll();

Why not move the creation of the StringBuilder to the setup() method and move its checking to the teardown() method? That is, provided that the assertSoft method is used by more than one test method.

Why not create some sort of assert utility class, instead of using StringBuilder?

@Test
public void f() {
SoftAssert sa = new SoftAssert();
sa.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled ");
sa.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find submit ");
sa.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find drpdwn1 ");
sa.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find radiobtn ");

sa.assertOk(); // whatever...
}

+1 for Sanjar's design - same impl as yours but just better design.

Similar design different names:

class AssertAccumulator {
void checkTrue(boolean expr) {...}
void assertAll() {...}
}

Also: instead of accumulating String messages perhaps use the actual assertTrue() in a try/catch and accumulate the AssertionExceptions -checkTrue(expr) would need to do this right away to catch stack/line correctly.

One solution (at least for Junit 4) is to write a runner class that checks the soft assertions. I have no experience with TestNG but I think this should be possible in TestNG.

@RunWith(SoftAssertRunner.class)
public class SoftAssertTest {
  @Test
  public void f(SoftAssert sa) {
    sa.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled ");
  }
}

The runner will check the assertion after each test method run.

Alternatively it could also be implemented with some ThreadLocal magic without an explicit SoftAssert parameter.

@RunWith(SoftAssertRunner.class)
public class SoftAssertTest {
  @Test
  public void f() {
    SoftAssert.assertTrue(selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled ");
  }
}

My 2 cents: Perhaps an assertAllTrue(boolean... all);

It goes through every one and on failure reports something like, "Error: Parameter 1 and 2 was false"

The limitation is there's no convenient way to add custom error messages for each param you pass in.

I worked with a proprietary testing system (developed before JUnit or TestNG existed) that had this concept. What it did was define a set of check* methods that had the same signature as the assert* methods but didn't stop the test if a failure happened. It also introduces a new assertion called assertChecks() which would stop the test if any checks had previously failed. Calling assertChecks() was optional, the framework would do it for you at the end of your test. This was very convenient and often used to provide tests that completely outlined their assertion errors without requiring lots of iterative runs.

I don't like the additional plumbing there - creating the StringBuilder at the beginning and calling assertOk() method at the end of the test method itself - those are things that should be handled by the framework.

What about adding a flag to the @Test annotation?

@Test (softAsserts = true)
public void verifyUI () {
... assertions ...
}

and let the framework handle any setup/teardown work in methods that run just before the method under test.

What about using Hamcrest?

assertThat(selenium, allOf(elementPresent("id=txtfiled"), elementPresent("id=submit"), ...))

Should this not already be possible by defining an aspect? You can create an around advice around each Assert statement that catches the assert exception and keeps it in a list or a StringBuilder. You also create an around advice around the actual test method to print out the list of assertions that have failed.

I agree with Alex, creating the StringBuilder at the beginning and calling assertOk() should be handled by the framework. I'm very keen to see soft assertions in TestNG.

I actually coded up most of this feature a couple of years ago and included it as a patch, but it never landed. It's associated with TESTNG-177 (your blogging software isn't letting me include the URL!)

Note that the solution provided in the original post is flawed because it's missing a key piece of functionality: you don't know the stacktrace of your soft failures.

@Lowell, that would be a big mistake for this feature! Failures in tearDown() cause the entire rest of the test class to be SKIPPED. Any "soft" assertion failure would prevent later tests from running.

What you want is a "verify" method instead, a la JIRA issue TESTNG-145.

I agree that this feature would be very usefull.
If you include it in the framework, it could manage the StringBuffer (no need to declare it and provide to each asserSoft call) by itself. It would be initialized for each runtime test, filled at each asserSoft call and emptied at each assertEmpty call. No need to add any flag to the Test annotation; no artefact to make it work but only 2 new methods: asserSoft and assertEmpty.
My 2 cents

I agree with Alex also, the details of how it should be handled (printing, store in a StrinBuilder, list, etcetera) should be hidden by the framework. Creating an annotation @RunSoft or parametrizing the existing one (@Test) are the best alternatives for me.

Personally I'd prefer to see the same syntax for the asserts with an annotation. That way you could write the entire thing using hard asserts, realize you needed them to be soft and then change it with a single @UseSoftAsserts.

That way you can also reuse the entire assert syntax rather than recreating each assert method with a "Soft" variant.

Hamcrest++

This would be incredibly useful, and I agree with Alex on the

"@Test (softAsserts = true)"

notation. It would be relatively easy to implement, and it would be incredibly useful. A lot of simple iterative tests that might currently require factories to run correctly could be done very simply with soft Asserts.

For EX: I am currently trying to test each object of an array of doubles against a converted array of doubles to be within a certain amount. It doesn't really justify the extra work of a factory, but if one double is within .0000001 of the threshold, I don't really care.

I'd vote for it being in the @Test tag too, just because you wouldn't have to rewrite each assert method, and it would be the easiest for people to implement.

The problem with changing all asserts to soft on a test by test basis is you wouldn't be able to mix hard and soft asserts. A lot of Selenium users do this. For example a title check would be a hard assert but certain page content checks might be soft.

Why not use "verify" instead of "assertSoft"? It is used in the default (junit3) SeleneseTestCase that comes with selenium, and the implementation is virtually identical.

The only difference being that the verify (soft assert) method wraps an actual assert in a try/catch block.

Hamcrest assertThat() doesn't improve on your initial assertTrue(a && b && c) except perhaps slightly in syntax.

I've implemented a verify (soft assertion) solution using an afterInvocation listener, based on the patch by Dan Fabiluch. I wrote about it on my blog here: seleniumexamples.com/blog/guide/using-soft-assertions-in-testng/