A question that has come up a few times already on the TestNG list is how to use asserts that won’t abort the entire test method.
Here is the latest example:
@Test public void verifyUi() { selenium.open (/* url */); assertTrue(selenium.isElementPresent("id=txtfiled")); assertTrue(selenium.isElementPresent("id=submit")); assertTrue(selenium.isElementPresent("id=drpdwn1")); assertTrue(selenium.isElementPresent("id=radiobtn")); }
The author would like each of these asserts to be run, even if one of them fails.
A simple solution to this problem is to use boolean tests to gather the state of each test and then have a final assert that will test them all:
boolean el1 = selenium.isElementPresent("id=txtfiled"); boolean el2 = selenium.isElementPresent("id=submit"); boolean el3 = selenium.isElementPresent("id=drpdwn1"); boolean el3 = selenium.isElementPresent("id=radiobtn"); assertTrue(el1 && el2 && el3 && el4);
The downside of this approach is that if the test fails, you won’t know exactly which element was not found on your web page, so we need to find a solution that will also provide a meaningful failure message.
Here is a more flexible way to approach this problem. It introduces a method assertSoft() that doesn’t throw any exception in the case of failure but that records the message if the test didn’t succeed. Error messages are accumulated in this variable, so it can contain more than one failure report.
Finally, a real assert method verifies that the error message is empty and throws an exception if it’s not:
public void assertSoft(boolean success, String message, StringBuilder messages) { if (!success) messages.append(message); } public void assertEmpty(StringBuilder sb) { if (sb.length() > 0) { throw new AssertionException(sb.toString()); } } @Test public void elementShouldBePresent() { StringBuilder result = new StringBuilder(); assertSoft(selenium.isElementPresent("id=txtfiled"), "Couldn't find txtfiled ", result); assertSoft(selenium.isElementPresent("id=txtfiled"), "Couldn't find submit ", result); assertSoft(selenium.isElementPresent("id=txtfiled"), "Couldn't find drpdwn1 ", result); assertSoft(selenium.isElementPresent("id=txtfiled"), "Couldn't find radiobtn ", result); assertEmpty(result); }
Can you think of a better way to solve this problem?
#1 by Sumit on June 25, 2009 - 10:52 am
A name like assertTrueButContinue might be more descriptive.
I also dislike having to manage StringBuilders separately. A chained way assertTrueButContinue( selenium.isElementPresent(“id=txtfiled”), “Couldn’t find txtfiled”).assertTrueButContinue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find submit”).summarize(); may be more consistent.
#2 by Sanjar Akhmedov on June 25, 2009 - 11:25 am
interface ComplexAssert {
ComplexAssert add(boolean success, String message);
void requireAll();
void requireAny();
}
ComplexAssert ca = …;
ca.add(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find txtfiled”)
.add(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find submit”)
.requireAll();
#3 by Lowell on June 25, 2009 - 11:41 am
Why not move the creation of the StringBuilder to the setup() method and move its checking to the teardown() method? That is, provided that the assertSoft method is used by more than one test method.
#4 by Tetsuo on June 25, 2009 - 12:52 pm
Why not create some sort of assert utility class, instead of using StringBuilder?
@Test
public void f() {
SoftAssert sa = new SoftAssert();
sa.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find txtfiled “);
sa.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find submit “);
sa.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find drpdwn1 “);
sa.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find radiobtn “);
sa.assertOk(); // whatever…
}
#5 by Sony Mathew on June 25, 2009 - 1:22 pm
+1 for Sanjar’s design – same impl as yours but just better design.
Similar design different names:
class AssertAccumulator {
void checkTrue(boolean expr) {…}
void assertAll() {…}
}
Also: instead of accumulating String messages perhaps use the actual assertTrue() in a try/catch and accumulate the AssertionExceptions -checkTrue(expr) would need to do this right away to catch stack/line correctly.
#6 by Thomas Jung on June 25, 2009 - 1:29 pm
One solution (at least for Junit 4) is to write a runner class that checks the soft assertions. I have no experience with TestNG but I think this should be possible in TestNG.
@RunWith(SoftAssertRunner.class)
public class SoftAssertTest {
@Test
public void f(SoftAssert sa) {
sa.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find txtfiled “);
}
}
The runner will check the assertion after each test method run.
Alternatively it could also be implemented with some ThreadLocal magic without an explicit SoftAssert parameter.
@RunWith(SoftAssertRunner.class)
public class SoftAssertTest {
@Test
public void f() {
SoftAssert.assertTrue(selenium.isElementPresent(“id=txtfiled”), “Couldn’t find txtfiled “);
}
}
#7 by tieTYT on June 25, 2009 - 1:32 pm
My 2 cents: Perhaps an assertAllTrue(boolean… all);
It goes through every one and on failure reports something like, “Error: Parameter 1 and 2 was false”
The limitation is there’s no convenient way to add custom error messages for each param you pass in.
#8 by Brandon on June 25, 2009 - 1:49 pm
I worked with a proprietary testing system (developed before JUnit or TestNG existed) that had this concept. What it did was define a set of check* methods that had the same signature as the assert* methods but didn’t stop the test if a failure happened. It also introduces a new assertion called assertChecks() which would stop the test if any checks had previously failed. Calling assertChecks() was optional, the framework would do it for you at the end of your test. This was very convenient and often used to provide tests that completely outlined their assertion errors without requiring lots of iterative runs.
#9 by Alex on June 25, 2009 - 2:29 pm
I don’t like the additional plumbing there – creating the StringBuilder at the beginning and calling assertOk() method at the end of the test method itself – those are things that should be handled by the framework.
What about adding a flag to the @Test annotation?
@Test (softAsserts = true)
public void verifyUI () {
… assertions …
}
and let the framework handle any setup/teardown work in methods that run just before the method under test.
#10 by Michael on June 25, 2009 - 7:28 pm
What about using Hamcrest?
assertThat(selenium, allOf(elementPresent(“id=txtfiled”), elementPresent(“id=submit”), …))
#11 by Wim Deblauwe on June 25, 2009 - 11:13 pm
Should this not already be possible by defining an aspect? You can create an around advice around each Assert statement that catches the assert exception and keeps it in a list or a StringBuilder. You also create an around advice around the actual test method to print out the list of assertions that have failed.
#12 by Dave Hunt on June 26, 2009 - 5:14 am
I agree with Alex, creating the StringBuilder at the beginning and calling assertOk() should be handled by the framework. I’m very keen to see soft assertions in TestNG.
#13 by Dan Fabulich on June 26, 2009 - 4:29 pm
I actually coded up most of this feature a couple of years ago and included it as a patch, but it never landed. It’s associated with TESTNG-177 (your blogging software isn’t letting me include the URL!)
Note that the solution provided in the original post is flawed because it’s missing a key piece of functionality: you don’t know the stacktrace of your soft failures.
#14 by Dan Fabulich on June 26, 2009 - 4:34 pm
@Lowell, that would be a big mistake for this feature! Failures in tearDown() cause the entire rest of the test class to be SKIPPED. Any “soft” assertion failure would prevent later tests from running.
What you want is a “verify” method instead, a la JIRA issue TESTNG-145.
#15 by Rod on June 27, 2009 - 1:18 am
I agree that this feature would be very usefull.
If you include it in the framework, it could manage the StringBuffer (no need to declare it and provide to each asserSoft call) by itself. It would be initialized for each runtime test, filled at each asserSoft call and emptied at each assertEmpty call. No need to add any flag to the Test annotation; no artefact to make it work but only 2 new methods: asserSoft and assertEmpty.
My 2 cents
#16 by Rafael Naufal on June 29, 2009 - 1:50 pm
I agree with Alex also, the details of how it should be handled (printing, store in a StrinBuilder, list, etcetera) should be hidden by the framework. Creating an annotation @RunSoft or parametrizing the existing one (@Test) are the best alternatives for me.
#17 by bill k on June 29, 2009 - 9:40 pm
Personally I’d prefer to see the same syntax for the asserts with an annotation. That way you could write the entire thing using hard asserts, realize you needed them to be soft and then change it with a single @UseSoftAsserts.
That way you can also reuse the entire assert syntax rather than recreating each assert method with a “Soft” variant.
#18 by Marvin Froeder on June 30, 2009 - 9:59 am
Hamcrest++
#19 by Mike R on July 30, 2009 - 7:49 am
This would be incredibly useful, and I agree with Alex on the
“@Test (softAsserts = true)”
notation. It would be relatively easy to implement, and it would be incredibly useful. A lot of simple iterative tests that might currently require factories to run correctly could be done very simply with soft Asserts.
For EX: I am currently trying to test each object of an array of doubles against a converted array of doubles to be within a certain amount. It doesn’t really justify the extra work of a factory, but if one double is within .0000001 of the threshold, I don’t really care.
I’d vote for it being in the @Test tag too, just because you wouldn’t have to rewrite each assert method, and it would be the easiest for people to implement.
#20 by Dave Hunt on August 26, 2009 - 3:47 am
The problem with changing all asserts to soft on a test by test basis is you wouldn’t be able to mix hard and soft asserts. A lot of Selenium users do this. For example a title check would be a hard assert but certain page content checks might be soft.
#21 by Aaron Evans on October 14, 2009 - 10:58 am
Why not use “verify” instead of “assertSoft”? It is used in the default (junit3) SeleneseTestCase that comes with selenium, and the implementation is virtually identical.
The only difference being that the verify (soft assert) method wraps an actual assert in a try/catch block.
Hamcrest assertThat() doesn’t improve on your initial assertTrue(a && b && c) except perhaps slightly in syntax.
#22 by Dave Hunt on October 24, 2009 - 7:44 am
I’ve implemented a verify (soft assertion) solution using an afterInvocation listener, based on the patch by Dan Fabiluch. I wrote about it on my blog here: seleniumexamples.com/blog/guide/using-soft-assertions-in-testng/
#23 by Subi on August 23, 2011 - 9:32 am
Hamcrest++
#24 by http://www.geekersmagazine.com/youtube-to-mp3/ on August 23, 2011 - 9:39 am
I liked your article Cedric. Thank you.
#25 by Nawaz on January 8, 2012 - 10:47 pm
Hi ,
I am facing a severe problem here.
I have four test, test1, test2 , test3 and test4
test2 is dependent on test1, test3 is dependent on test2 and test 4 on test3. I am running my test using ANT by adding testNG task in ANT File (build.xml).
My Problem If any failure occur in test1 i.e . in Soft Assertion Then Test2, Tes3 and Test4 gets skipped that it they are not executed. I want them to execute.
My Requirement is simple I want to execute the dependent methods if soft assertions occurs and does not want to execute dependent methods if failure occurs in hard Assertions.
Please help me