Handling multiple failures in single test in Junit 4

In my last post we have seen handling multiple assertion failures in single test using junit 3.x. In this post we’ll see how to overcome the same problem but in junit 4 style.

Junit 4 addresses the problem of handling multiple assertion failures pretty well. It has an ErrorCollector class that collects all failures in a test case and reports them at the end of test case. The trick lies in usage of error collector which is also not much difficult.

All you have to do is follow the below mentioned changes and update your base class accordingly.

1. Let’s assume you have a Base class which is extended by all your test classes. In my example it’s the TestBase.java. In the base class create an ErrorCollector object and (most important) add @Rule annotation to it.
2. Now create a Boolean variable “fatal” and add setter and getter methods for it. This will help you specify which test cases you want to make fatal and which you don’t.
3. Create an assertEquals(String, Object, Object) method with public access. We will be using this method in all our test classes that will extend the TestBase.java. This method will be the decision maker on how the test execution should go ahead (whether fatal or non-fatal).

Your code should look like the one mentioned below:

public class TestBase {

      @Rule

      public ErrorCollector collector = new ErrorCollector();

     

      private boolean fatal;

     

      public TestBase() {

            fatal=true;

      }

     

      public void assertEquals(String msg, Object expected, Object actual) {

            if(getFatal()) {

                  Assert.assertEquals(msg,expected, actual);

            } else {

                  collector.checkThat(msg, actual, CoreMatchers.equalTo(expected));

            }

      }

     

      public void setFatal(boolean fatalFlag) {

            fatal = fatalFlag;

      }

     

      public boolean getFatal() {

            return fatal;

      }

     

}

That’s all!!!

You are done with all the implementation part. Now, all you have to do is to use it, and that is demonstrated below. The test class TestSample.java extends the TestBase.java to make use of the code implemented in it.

public class TestSample extends TestBase{

     

      @Test

      public void one() {

            assertEquals("Comparing 1 and 2", 1, 2);

            assertEquals("Comparing A and B", "A", "B");

            assertEquals("Comparing 10.2 and 10.2", 10.2, 10.2);

      }

     

      @Test

      public void two() {

            setFatal(false);

            assertEquals("Comparing 1 and 2", 1, 2);

            assertEquals("Comparing A and B", "A", "B");

            assertEquals("Comparing 10.2 and 10.2", 10.2, 10.2);

      }

}

The output will be: Runs 2; Failures:3; Error:0

1. For test case “one”:
   java.lang.AssertionError: Comparing 1 and 2 expected:<1> but was:<2>

2. For test case “two”:
   java.lang.AssertionError: Comparing 1 and 2
   Expected: <1> 
        got: <2>

java.lang.AssertionError: Comparing A and B

Expected: "A"

     got: "B"

As you can see that for test case “one” which is by default fatal (default junit behabiour), the test case exited after the first assertion failure.

Whereas, for test case “two” which is explicitly marked as non-fatal by setting the fatal flag to false, all the assertions within the test case were executed even though the first assertion failed.

So, by doing few changes in your base class of your test framework, you can add an important feature of handling multiple assertion failures to your framework.

That’s all for this post.

I’ll keep posting up some interesting junit problems on this blog, as and when I encounter (and solve) them.

Feel free to comment if you like the post or have any good suggestions.

Junit 3 - Handling multiple failures in single test

When testers use junit framework for functional testing, one feature they desperately need is to continue the test execution even when one or more assertions fail in a test case.

If you are using junit 3 framework for functional testing then you can use this feature by using “Jfunc” jar.

By default in junit all the tests are fatal i.e. when an assertion fails in a test case, the code following that assertion is not executed. But Jfunc gives you a choice to control this; you can either continue to execute all the code of your test-case even when an assert fails, or to stop it after the first failure itself.

Changes to your framework to use Jfunc.jar

To start using Jfunc, all you need to do is to extend your Test Class to junit.extensions.jfunc.JFuncTestCase base class instead of junit.framework.TestCase, and that’s all. Rest everything (constructor initialization and suite method) remains the same as you do it in Junit framework.

Making a test non-fatal

Now comes the main part; how to make a test execution non-fatal.

This is also very simple, all you have to do is, just set the fatal flag to false (setFatal(false)) in your test class’s constructor and all the tests in your test class will be non-fatal.

An example of using Jfunc in your test class is mentioned below.

public class TestClass extends JFuncTestCase { 

     

      public TestClass(String name) {

            super(name);

            setFatal(false);        

      }

     

      public static Test suite() {

            return new TestSuite(TestClass.class);

      }

     

      public void test1() {

            fail("test1: fail 1");

            fail("test1: fail 2");

            fail("test1: fail 3");

      }

     

      public void test2() {

            fail("test2: fail 1");

            fail("test2: fail 2");

            fail("test2: fail 3");

      }

}

The best part of using Jfunc’s fatal feature is that you can set it at test case level also.

For example, in the above code if I want to make only test1 as non-fatal (execute all fail() methods) and I want that test2 should be fatal (stop execution after first fail() method).

Then all I have to do is to remove the setFatal(false) from the constructor and add it at test case level as shown below.

      public TestClass(String name) {

            super(name);

//          setFatal(false);

      }

     

      public void test1() {

            setFatal(false);       

            fail("test1: fail 1");

            fail("test1: fail 2");

            fail("test1: fail 3");

      }

     

      public void test2() {

            fail("test2: fail 1");

            fail("test2: fail 2");

            fail("test2: fail 3");

      }

Now, only test1 will have a non-fatal behavior and all other tests of TestClass.java will be fatal.

The last but not the least; by default the fatal flag is set to true i.e. if you did not use setFatal() method anywhere inside your test class, then, all your test cases will be fatal (junit style).

Conclusion:

Since the release of junit 4.x (recent version is 4.10) with a variety of good features (see my last post for junit 4 features), most of the junit testers are planning or had already migrated to the new junit 4.x framework.

Jfunc internally uses Junit 3.x, and does not have support for junit 4.x. So, if you have used jfunc and its fatal feature in your framework and planning to migrate to junit 4.x, then the biggest challenge for you is to have the Jfunc’s fatal feature in the 4.x framework.

In my next post we’ll see how we can achieve this using junit 4.10 framework.

Migrating from Junit 3.x to Junit 4.x

Hello and Welcome,

This is my first blog, so thought to start with something I recently did; and it was test framework migration from junit 3.x to junit 4.x.
Junit 4.x has now came up with variety of good features that gives functional or API testers flexibility to write better tests for example; you can put a timeout time for a test case, you can parametrized a test case, etc.

As like other technologies, Junit 4.x is also backward compatible with its previous versions (i.e. junit 3.x). So a code that is written in junit 3.x can anytime be executed on junit 4.x as well. But if you want to use junit 4.x in real sense (and not just a jar that runs 3.x code as well) I would encourage you to migrate your junit 3.x code to junit 4.x and start using features (powers) of junit 4.x.

So, following are the steps which you need to follow while migrating your code from junit 3.x to junit 4.x

1.       Remove all references of junit 3.x jar from the classpath and instead add junit 4.x jar.

As I told junit 4.x is backward compatible with junit 3.x so after adding junit 4.x jar, your code will not throw any compile error and hence you need to be more careful to make sure that you have removed all references of  junit 3.x.

At any point if you find “junit.framework.XXX” references in your test framework code then go ahead and remove it.

2.       Once the junit 4.x jar is added, go to your test base class (a class that is extended by all test suites in your framework). The test base class would be extending junit.framework.TestCase; remove this reference (delete extends TestCase part). If you have individual test suites extending junit.framework.TestCase then remove it from the individual test suites.

The point is junit.framework.TestCase reference should be removed; nothing in your framework should extend this class.

3.       In your test suite if you have a suite method, add @BeforeClass annotation above it. Basically, it is good idea to rename the basic junit 3.x standard methods (like suite(), setup(), teardown(), etc) just to be more sure that nothing is going junit 3.x way; but if you have taken care in removing all junit 3.x imports, then its ok to have the same method names.

4.       If you have a setup() and teardown() methods in your suite and/or test base class, then add @Before and @After annotations to it respectively. Again, you can rename these methods as well or keep them as is it is if you are very sure that all junit 3.x imports are removed.

The important thing here is that you can add @BeforeClass to only one method (per class) but you can add @Before and @After annotations to more than one methods.

5.       All the asserts in your test class/suite should come from class org.junit.Assert. You need to add following statement either to your base class (if you are overriding junit asserts in your base class) or to the individual test class/suite if you don’t have any base class in your framework.

import static org.junit.Assert.*;

6.       Finally, add @Test annotation to all your test cases.

If you want to add more than one suite to your test class, then this can be done using @RunWith and @SuiteClasses annotations.

For example: if you have a code in junit 3.x style similar to the one mentioned below;

Class MyTestPackage

{

public static Test suite()

{

TestSuite suite = new TestSuite();

suite.addTest(TestOne.suite());

suite.addTest(TestTwo.suite());

return suite;

}

}

then this will be migrated to junit 4.x as

@RunWith(Suite.class)

@SuiteClasses({

      TestOne.class,

      TestTwo.class })

Class MyTestPackage

{

}

In most of the cases the testers migrating to junit 4.x will have lots of test classes/suites that need to be refactored using the above mentioned steps. Sometimes, the count can be in hundreds.

So, if you have few test classes in your framework (may be 10 or 20) you can go ahead and simply apply the above mentioned changes to those. But if there are large numbers of test classes/suites in your framework, it’s better to use a junit 3.x to junit 4.x convertor tool. There are many open source tools that are available on internet that help in automatically migrating from junit 3.x to junit 4.x.

In my next post I'll be coming up with a common feature that many junit testers demand, and that is, continue test execution even when one of the asserts fail in the test case and report the failing asserts in the test execution report. We'll see that this can be achieved with both, junit 3.x as well as junit 4.x.