May 5, 2008

Multithreaded Testing

Every now and then you'll work on something that needs to handle requests from multiple concurrent threads in a special way. I say "special way" because in a web application, everything needs to handle being executed concurrently and there are a slew of techniques used to handle this (prototypes, thread locals, stateless services, etc). Here's an example of what I mean by "special"...

On my current project, we have a queue of articles that need human-user attention. Each article must be doled out to only one user and there are multiple instances of the web application servicing requests in the cluster. We can't rely on Java synchronization because it only works within the JVM instance, not across instances.

The simplified version of the service interface we're working on looks like this:

public interface ArticleService {
 
     Article findNextArticleForModeration();
 
}

What makes this interesting is that we must ensure that the service doesn't hand out the same Article to more than one user. This is impossible to assert using a single thread. We've all been told that multiple threads and automated testing don't mix. It's generally true and should be avoided if at all possible, but in some cases it's the only way we can truly assert specific behavior. I've found a pretty simple way to do this type of testing in a reliable, consistent, and non-disruptive manner. Despite the fact that the technique leverages Java 1.5 built-in concurrency utilities, most of the engineers who have seen it are surprised and weren't aware that such testing was so easy to implement.

Given the above service interface, here's a test that will assert that no single article is given out to more than one invoker of the method findNextArticleForModeration(). The scenario we're simulating is 10 users feverishly moderating a queue of 250 articles as quickly as possible.

public void findNextArticleForModerationStressTest() throws Exception
{
     final int ARTICLE_COUNT = 250;
     final int THREAD_COUNT = 10;
 
     // Create test data and callable tasks
     //
     Set<Article> testArticles = new HashSet<Article>();
 
     Collection<Callable<Article>> tasks = new ArrayList<Callable<Article>>();
     for (int i = 0; i < ARTICLE_COUNT; i++)
     {
          // Test data
          testArticles.add(new Article());
  
          // Tasks - each task makes exactly one service invocation.
          tasks.add(new Callable<Article>()
          {
               public Article call() throws Exception
               {
                    return articleService.findNextArticleForModeration();
                }
           });
      }
     articleService.createArticles(testArticles);
 
     // Execute tasks
     //
     ExecutorService executorService = Executors.newFixedThreadPool(THREAD_COUNT);
     // invokeAll() blocks until all tasks have run...
     List<Future<Article>> futures = executorService.invokeAll(tasks);    
     assertThat(futures.size(), is(ARTICLE_COUNT));
 
     // Assertions
     //
     Set<Long> articleIds = new HashSet<Long>(ARTICLE_COUNT);
     for (Future<Article> future : futures)
     {
          // get() will throw an exception if an exception was thrown by the service.
          Article article = future.get();
          // Did we get an article?
          assertThat(article, not(nullValue()));
          // Did the service lock the article before returning?
          assertThat(article.isLocked(), is(true));
          // Is the article id unique (see Set.add() javadoc)?
          assertThat(articleIds.add(article.getId()), is(true));
      }
     // Did we get the right number of article ids?
     assertThat(articleIds.size(), is(ARTICLE_COUNT));
}

The test starts off by creating 250 test articles to be moderated. It also creates 250 'tasks', each designed to make a single service invocation of findNextArticleForModeration(). The real magic happens in Executors.newFixedThreadPool() and executorService.invokeAll(). The first creates a new ExecutorService backed by a thread pool of the specified size. This is a generic ExecutorService that is designed to churn through tasks using all of the threads in the pool. invokeAll blocks until every task has finished executing. In this test, 10 threads will rip through 250 tasks, each making a single call to our service and capturing the result of that call. Each task execution results in a Future, which is a handle to the results of the task (and more).

Iterating over each resulting future, we make several assertions. The most important one is the last, where we assert that every task is given a unique Article. Thanks to the natural semantics of Set, this is easy to do in an elegant way. Another useful, though unexpected, feature is that if an exception occurs during the task execution, an ExecutionException will be thrown when get() is called on the corresponding Future. If our service fails for some reason, the test will fail because no exceptions are expected.

This technique makes simulating a multi-threaded environment in a test easy and readable. It's important to only use this technique when it's really necessary. The resulting test is more of an integration test than a unit test, and its run time is an order of magnitude or more than a unit test, so overuse of the technique will artificially inflate the time it takes to runs the tests. After I've finished working on the component under test, I will reduce the test-data size and thread count to a level that the test still provides value, but is no longer a stress test (e.g. 10 articles and 2 threads). The next time the component is being worked on, the developer can crank up the values and run the tests to be confident that the behavior isn't broken.

The complete source for a working example of this technique is available here. You'll need Maven (or IntelliJ IDEA 7.x) to build and run the test. By default, the tests run against an in-memory H2Database instance, but if you look at application.properties you'll see configurations for PostgreSQL and MySQL as well.

Happy testing!

Posted by Christian Nelson at 3:06 PM | Comments (0)

May 15, 2007

DBUnit 2.2, Spring, and Testing

You may have noticed that DBUnit changed its connection closing behavior in v2.2. We noticed it when tests deriving from our custom DatabaseTestCase implementation (which uses DBUnit) starting failing. Initially we just reverted back to version 2.1. Since then I've discovered what the problem was and found a workaround, and while I was at it, I created some nice utilities to be used in tests which need data fixtures.

Problem: In v2.2 DBUnit introduces some new abstractions, one of which is the IDatabaseTester. Whether it’s by design or by accident, the default implementation closes connections after executing its operations (more info). The end result is that in our unit tests, the database connection is closed before the test can run.

Solution: I’ve created two simple util classes for loading test data into a database without closing the connection:

/** A helper for loading data sets into unit tests. */
public class DatabaseUtils {
     public static void loadDataSet(Class clazz, final DataSource dataSource) throws Exception {
          IDataSet dataSet = new FlatXmlDataSet(TestUtils.datasetInputStream(clazz));
          IDatabaseTester tester = new ExistingConnectionDatabaseTester(dataSource);
          tester.setDataSet(dataSet);
          tester.onSetup();
      }
}

 
/** A special DatabaseTester that doesn’t close the connection when its done. */
public class ExistingConnectionDatabaseTester extends AbstractDatabaseTester {
     private DataSource dataSource;
  
     public ExistingConnectionDatabaseTester(DataSource dataSource) {
          super();
          this.dataSource = dataSource;
      }
  
     public IDatabaseConnection getConnection() throws Exception {
          return new DatabaseConnection(DataSourceUtils.getConnection(dataSource));
      }
  
     public void closeConnection(IDatabaseConnection connection) throws Exception {
          // Don't close that connection!
      }
}

The first of these depends on another test-related utility, TestUtils.java (see below), which converts a class to a path. The second depends on Spring’s DataSourceUtils.

 
/**
 * A set of utilities that generate paths from classnames.  This is useful when
 * making reference to resources used by test cases, when the resources are
 * located in a directory path matching the class' package.
 */
public class TestUtils {
 
     public static final String TEST_PREFIX = "src/test/resources/"; // Maven2 default
 
     public static String pathString(Class clazz) {
          return pathString(TEST_PREFIX, clazz, "");
      }
 
     public static String pathString(Class clazz, String resource) {
          return pathString(TEST_PREFIX, clazz, resource);
      }
 
     public static String pathString(String prefix, Class clazz, String resource) {
          prefix = (prefix != null ? prefix : "");
  
          StringBuffer sb = new StringBuffer();
          sb.append(prefix);
  
          if (!prefix.endsWith("/")) {
               sb.append("/");
           }
  
          sb.append(ClassUtils.classPackageAsResourcePath(clazz));
          sb.append("/");
          sb.append(resource);
  
          return sb.toString();
      }
 
     public static InputStream pathInputStream(Class clazz, String resource) throws FileNotFoundException {
          return pathInputStream(TEST_PREFIX, clazz, resource);
      }
 
     public static InputStream pathInputStream(String prefix, Class clazz, String resource) throws FileNotFoundException {
          return new FileInputStream(pathString(prefix, clazz, resource));
      }
 
     public static InputStream datasetInputStream(Class clazz) throws FileNotFoundException {
          return pathInputStream(TEST_PREFIX, clazz, ClassUtils.getShortName(clazz) + ".xml");
      }
}

These three classes give us everything we need to load data into our test database using DBUnit. And TestUtils can be used to load other test resources from the classpath as well, like images, documents, CSVs, etc.

Here's an example of how it can be used in a DAO test based on Spring's test classes.

public class ArtistHibernateDaoTest extends AbstractAnnotationAwareTransactionalTests {
  
     private ArtistHibernateDao artistDao;
  
     public void setArtistDao(ArtistHibernateDao artistDao) {
          this.artistDao = artistDao;
      }
  
     protected String[] getConfigLocations() {
          return new String[]{"applicationContext-database.xml", "applicationContext-hibernate.xml"};
      }
  
     protected void onSetUpInTransaction() throws Exception {
          DatabaseUtils.loadDataSet(getClass(), getJdbcTemplate().getDataSource());
      }
  
     public void testFindAllNames() {
          List<String> names = artistDao.findAllNames();
          assertEquals(3, names.size());
          assertTrue(names.contains("The Cure"));
          assertTrue(names.contains("Depeche Mode"));
          assertTrue(names.contains("New Order"));
      }
}

onSetUpInTransaction() loads the data set using the convention of (packagename).(ClassName).xml before each test (e.g. com.c5.PizzaTest loads the file com/c5/PizzaText.xml on the classpath). In this case the data is loaded in the same transaction as the test so that no clean-up is necessary when the test is completed. These tests run fast!

We've been talking about moving away from our custom DatabaseTestCase class hierarchy. The above functionality in conjunction with Spring's test hierarchy could be a good start.

On a somewhat related note, there is a new unit testing framework called Unitils which does this sort of thing and much more. I haven’t used it but it sounds interesting.

Christian

Posted by Christian Nelson at 9:30 AM | Comments (0)

May 6, 2007

Generic Custom Argument Matching in EasyMock

EasyMock is a clean, simple library for creating mock objects. It provides a whole range of facilities for declaring our expectations of the method call interactions on the mock including call count, call order, return values, exceptions, and argument matchers. Unfortunately, the argument matchers provide a limited set of argument expectations and a somewhat cumbersome process for expanding that set. Using java generics, it is possible to create a custom argument matcher allowing simple expression of unlimited assertions about Object arguments.

Imagine we have a test fixture, BlogEntryController, which depends on a single service, BlogService. Because we want to focus our testing on the fixture, we'll inject a mock object as the service.

//follow EasyMock's convention of static imports
import static org.easymock.EasyMock.*
...
BlogEntryController controller = new BlogEntryController();
BlogService mockBlogService = createMock(BlogService.class);
controller.setBlogService(mockBlogService);

The next step is to record the expectations about the interactions between the fixture and its collaborator. In the record phase, we simply interact with the mock to record these expectations:

//record phase
BlogEntry blogEntry = new BlogEntry();
mockBlogService.saveBlogEntry(blogEntry);

//replay phase
replay(mockBlogService);

//method under test
controller.handleRequest(request, response);

//verify
verify(mockBlogService);

By default, argument expectations are verified using the equals() method. For the previous expectation, when the controller code is run, EasyMock will assert that the BlogEntry passed in by the controller equals() that passed in during the record phase. If, for instance, the controller creates its own BlogEntry and BlogEntry has the default implementation of equals(), this expectation would fail. Fortunately, EasyMock provides other argument matchers which can verify arguments based on different criteria:

mockBlogService.saveBlogEntry(isA(BlogEntry.class));

By using the isA(Class clazz) argument matcher, one of several predefined in EasyMock, the code will verify that the argument passed from the controller to the mock service is an instance of the BlogEntry class. By relaxing our expectations, we've come up with a test case which would pass. However, generally speaking, we'll probably want to test some more details about the BlogEntry. For instance, it's common to want to verify the argument's properties.

EasyMock has the facility to create custom argument matchers. It's a somewhat heavy weight solution requiring the implementation of 3 methods: 2 for the IArgumentMatcher interface and one static method used during the record phase. This facility seems better suited to creating reusable argument matchers (akin to the predefined ones which come with EasyMock) rather than singular matchers relevant only to a particular test case.

To bridge the gap, I have created a generic argument matcher, EasyMockUtils.argAssert(Assertion<E> assertion), which abstracts away the required EasyMock interaction code, allowing one to more simply express, inline if desired, the argument expectations:

static import common.test.easymock.EasyMockUtils;
...
mockBlogService.saveBlogEntry(argAssert(new Assertion<BlogEntry>() {
   void check(BlogEntry blogEntry) {
      assertEquals("submitted blog entry body", blogEntry.getBody());
      assertNull(blogEntry.getDateCreated());
    })
}

The argAssert static method takes a genericized instance of the Assertion interface in which one can declare any number of assertions against the argument passed in during the replay phase. As an added benefit, the Assertion implementation could also be leveraged to modify the passed in argument. For instance, imagine the controller depends on the service setting the dateCreated property on the passed in bean. We can ensure our mock replicates that behavior:

mockBlogService.saveBlogEntry(argAssert(new Assertion<BlogEntry>() {
   void check(BlogEntry blogEntry) {
      assertEquals("submitted blog entry body", blogEntry.getBody());
      assertNull(blogEntry.getDateCreated());
      //mimic the behavior of the true service
      blogEntry.setDateCreated(new Date());
    })
}

Mocking to this detail could trigger a warning regarding the level of coupling between the test, the fixture, and the collaborator; however, I have seen situations where it can make the difference between being able to use a mock object for testing or not. Mockist testers in general must be wary of unnecessarily deep coupling where, for instance, a refactor of the fixture/collaborator interactions could result in as much time spent updating test code even if the test's high level assertions remain unchanged. However, used attentively during test driven development, mock objects can help a programmer focus on keeping those interactions simple and clearly defined. Furthermore, testing with mock objects helps one to focus on unit testing the fixture at hand. In that situation, the generic custom argument matcher provides a simplified tool for expressing assertions about that code.

And here's the code. The generic interface which allows one to specify argument assertions:

Assertion.java

package common.test.easymock;

public interface Assertion<E>
{
   /**
    * Allows specifying varying assertions about an argument.
    * @param argument the argument being check
    * @throws Error throw an exception in the case of an assertion failure
    */
   void check(E argument) throws Error, Exception;
}

A simple container for the static method used to declare the argument matching assertion during the mock record phase.

EasyMockUtils.java

package common.test.easymock;

import org.easymock.EasyMock;

public class EasyMockUtils
{
   public static <E> E argAssert(Assertion<E> assertion)
   {
      EasyMock.reportMatcher(new ArgumentAssertion(assertion));
      return null;
    }
}

And the IArgumentMatcher implementation which ties the pieces together:

ArgumentAssertion.java

package common.test.easymock;

import org.easymock.IArgumentMatcher;

public class ArgumentAssertion implements IArgumentMatcher {
   private Assertion assertion;
   private Error assertionError;
 
   public ArgumentAssertion(Assertion assertion) {
      this.assertion = assertion;
    }
 
   public boolean matches(Object actual) {
      try
      {
         assertion.check(actual);
         return true;
       }
      catch (Error e)
      {
         assertionError = e;
         return false;
       }
      catch (Exception e)
      {
         assertionError = new Error(e);
         return false;
       }
    }
 
   public void appendTo(StringBuffer buffer) {
      buffer.append("argumentAssertion(exception ").append(assertionError).append(")");
    }
}

And of course a test:

ArgumentAssertionTest.java

package common.test.easymock;

import static common.test.easymock.EasyMockUtils.argAssert;
import junit.framework.TestCase;
import static org.easymock.EasyMock.*;

public class ArgumentAssertionTest extends TestCase
{
   public interface TestInterface
   {
      void testMethod(String arg);
    }
 
   TestInterface mock;
 
   protected void setUp() throws Exception
   {
      super.setUp();
      mock = createMock(TestInterface.class);
    }
 
   public void testArgumentAssertionSucess()
   {
      //record expected behavior
      mock.testMethod(argAssert(new Assertion<String>() {
         public void check(String argument)
         {
            //do nothing - should pass
          }
       }));
  
      //should get no errors
      replay(mock);
      mock.testMethod("test");
      verify(mock);
    }
 
   public void testArgumentAssertionFailure()
   {
      //record expected behavior
      mock.testMethod(argAssert(new Assertion<String>() {
         public void check(String argument)
         {
            fail("explicitely fail assertion");
          }
       }));
  
      replay(mock);
  
      boolean testSucceeded = false;
  
      try
      {
         mock.testMethod("test");
         verify(mock);
         testSucceeded = true;
       }
      catch (AssertionError e)
      {
         //should get this error
       }
  
      if (testSucceeded)
        fail("should have gotten a failure");
    }
}

Posted by Randy Puro at 12:05 PM | Comments (0)

July 20, 2005

DBUnit, HSQL and the BOOLEAN data type

We have been using HSQL in-memory along with DBUnit for unit testing lately, and I found an issue using the most recent version of each. Basically, HSQL has added a new data type, BOOLEAN, which replaces BIT. But DBUnit is not updated to support this, and an error is throw when you attempt to insert some BOOLEAN data using DBUnit.

The error looks like this:

WARNING - TABLE.COLUMN data type (16, 'BOOLEAN') not recognized 
and will be ignored. See FAQ for more information.

The solution outlined here is straightforward. You need to create a new data type factory that extends the DBUnit class DefaultDataTypeFactory. This new class just handles the SQL type Types.BOOLEAN as a special case. The code follows:

HsqlDataTypeFactory.java

package company.project.test;

import org.apache.commons.logging.*;
import org.dbunit.dataset.datatype.*;

import java.sql.*;

public class HsqlDataTypeFactory
  extends DefaultDataTypeFactory
{
   private static final Log log = LogFactory.getLog(HsqlDataTypeFactory.class);
 
   public DataType createDataType(int sqlType, String sqlTypeName)
     throws DataTypeException
   {
      if (sqlType == Types.BOOLEAN)
      {
         return DataType.BOOLEAN;
       }
  
      return super.createDataType(sqlType, sqlTypeName);
    }
}

Then, in order to use this data type factory, just set a property on the IDatabaseConnection DBUnit object in your code (here is an example method):

 protected IDatabaseConnection getConnection() throws Exception
  {
     IDatabaseConnection connection = 
       new DatabaseConnection( dataSource.getConnection() );
     DatabaseConfig config = connection.getConfig();
     config.setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, 
                        new HsqlDataTypeFactory());
 
     return connection;
   }

Posted by Mike Wynholds at 10:40 AM | Comments (5)