Sunday 22 July 2012

Junit 3 - Handling multiple failures in single test


When testers use junit framework for functional testing, one feature they desperately need is to continue the test execution even when one or more assertions fail in a test case.
If you are using junit 3 framework for functional testing then you can use this feature by using “Jfunc” jar.
By default in junit all the tests are fatal i.e. when an assertion fails in a test case, the code following that assertion is not executed. But Jfunc gives you a choice to control this; you can either continue to execute all the code of your test-case even when an assert fails, or to stop it after the first failure itself.

Changes to your framework to use Jfunc.jar

To start using Jfunc, all you need to do is to extend your Test Class to junit.extensions.jfunc.JFuncTestCase base class instead of junit.framework.TestCase, and that’s all. Rest everything (constructor initialization and suite method) remains the same as you do it in Junit framework.

Making a test non-fatal

Now comes the main part; how to make a test execution non-fatal.
This is also very simple, all you have to do is, just set the fatal flag to false (setFatal(false)) in your test class’s constructor and all the tests in your test class will be non-fatal.
An example of using Jfunc in your test class is mentioned below.

public class TestClass extends JFuncTestCase { 
     
      public TestClass(String name) {
            super(name);
            setFatal(false);        
      }
     
      public static Test suite() {
            return new TestSuite(TestClass.class);
      }
     
      public void test1() {
            fail("test1: fail 1");
            fail("test1: fail 2");
            fail("test1: fail 3");
      }

     
      public void test2() {
            fail("test2: fail 1");
            fail("test2: fail 2");
            fail("test2: fail 3");
      }
}

The best part of using Jfunc’s fatal feature is that you can set it at test case level also.
For example, in the above code if I want to make only test1 as non-fatal (execute all fail() methods) and I want that test2 should be fatal (stop execution after first fail() method).
Then all I have to do is to remove the setFatal(false) from the constructor and add it at test case level as shown below.

      public TestClass(String name) {
            super(name);
//          setFatal(false);
      }
     
      public void test1() {
            setFatal(false);       
            fail("test1: fail 1");
            fail("test1: fail 2");
            fail("test1: fail 3");
      }

     
      public void test2() {
            fail("test2: fail 1");
            fail("test2: fail 2");
            fail("test2: fail 3");
      }

Now, only test1 will have a non-fatal behavior and all other tests of TestClass.java will be fatal.
The last but not the least; by default the fatal flag is set to true i.e. if you did not use setFatal() method anywhere inside your test class, then, all your test cases will be fatal (junit style).

Conclusion:

Since the release of junit 4.x (recent version is 4.10) with a variety of good features (see my last post for junit 4 features), most of the junit testers are planning or had already migrated to the new junit 4.x framework.
Jfunc internally uses Junit 3.x, and does not have support for junit 4.x. So, if you have used jfunc and its fatal feature in your framework and planning to migrate to junit 4.x, then the biggest challenge for you is to have the Jfunc’s fatal feature in the 4.x framework.
In my next post we’ll see how we can achieve this using junit 4.10 framework.