May 5, 2008

Multithreaded Testing

Every now and then you'll work on something that needs to handle requests from multiple concurrent threads in a special way. I say "special way" because in a web application, everything needs to handle being executed concurrently and there are a slew of techniques used to handle this (prototypes, thread locals, stateless services, etc). Here's an example of what I mean by "special"...

On my current project, we have a queue of articles that need human-user attention. Each article must be doled out to only one user and there are multiple instances of the web application servicing requests in the cluster. We can't rely on Java synchronization because it only works within the JVM instance, not across instances.

The simplified version of the service interface we're working on looks like this:

public interface ArticleService {
 
     Article findNextArticleForModeration();
 
}

What makes this interesting is that we must ensure that the service doesn't hand out the same Article to more than one user. This is impossible to assert using a single thread. We've all been told that multiple threads and automated testing don't mix. It's generally true and should be avoided if at all possible, but in some cases it's the only way we can truly assert specific behavior. I've found a pretty simple way to do this type of testing in a reliable, consistent, and non-disruptive manner. Despite the fact that the technique leverages Java 1.5 built-in concurrency utilities, most of the engineers who have seen it are surprised and weren't aware that such testing was so easy to implement.

Given the above service interface, here's a test that will assert that no single article is given out to more than one invoker of the method findNextArticleForModeration(). The scenario we're simulating is 10 users feverishly moderating a queue of 250 articles as quickly as possible.

public void findNextArticleForModerationStressTest() throws Exception
{
     final int ARTICLE_COUNT = 250;
     final int THREAD_COUNT = 10;
 
     // Create test data and callable tasks
     //
     Set<Article> testArticles = new HashSet<Article>();
 
     Collection<Callable<Article>> tasks = new ArrayList<Callable<Article>>();
     for (int i = 0; i < ARTICLE_COUNT; i++)
     {
          // Test data
          testArticles.add(new Article());
  
          // Tasks - each task makes exactly one service invocation.
          tasks.add(new Callable<Article>()
          {
               public Article call() throws Exception
               {
                    return articleService.findNextArticleForModeration();
                }
           });
      }
     articleService.createArticles(testArticles);
 
     // Execute tasks
     //
     ExecutorService executorService = Executors.newFixedThreadPool(THREAD_COUNT);
     // invokeAll() blocks until all tasks have run...
     List<Future<Article>> futures = executorService.invokeAll(tasks);    
     assertThat(futures.size(), is(ARTICLE_COUNT));
 
     // Assertions
     //
     Set<Long> articleIds = new HashSet<Long>(ARTICLE_COUNT);
     for (Future<Article> future : futures)
     {
          // get() will throw an exception if an exception was thrown by the service.
          Article article = future.get();
          // Did we get an article?
          assertThat(article, not(nullValue()));
          // Did the service lock the article before returning?
          assertThat(article.isLocked(), is(true));
          // Is the article id unique (see Set.add() javadoc)?
          assertThat(articleIds.add(article.getId()), is(true));
      }
     // Did we get the right number of article ids?
     assertThat(articleIds.size(), is(ARTICLE_COUNT));
}

The test starts off by creating 250 test articles to be moderated. It also creates 250 'tasks', each designed to make a single service invocation of findNextArticleForModeration(). The real magic happens in Executors.newFixedThreadPool() and executorService.invokeAll(). The first creates a new ExecutorService backed by a thread pool of the specified size. This is a generic ExecutorService that is designed to churn through tasks using all of the threads in the pool. invokeAll blocks until every task has finished executing. In this test, 10 threads will rip through 250 tasks, each making a single call to our service and capturing the result of that call. Each task execution results in a Future, which is a handle to the results of the task (and more).

Iterating over each resulting future, we make several assertions. The most important one is the last, where we assert that every task is given a unique Article. Thanks to the natural semantics of Set, this is easy to do in an elegant way. Another useful, though unexpected, feature is that if an exception occurs during the task execution, an ExecutionException will be thrown when get() is called on the corresponding Future. If our service fails for some reason, the test will fail because no exceptions are expected.

This technique makes simulating a multi-threaded environment in a test easy and readable. It's important to only use this technique when it's really necessary. The resulting test is more of an integration test than a unit test, and its run time is an order of magnitude or more than a unit test, so overuse of the technique will artificially inflate the time it takes to runs the tests. After I've finished working on the component under test, I will reduce the test-data size and thread count to a level that the test still provides value, but is no longer a stress test (e.g. 10 articles and 2 threads). The next time the component is being worked on, the developer can crank up the values and run the tests to be confident that the behavior isn't broken.

The complete source for a working example of this technique is available here. You'll need Maven (or IntelliJ IDEA 7.x) to build and run the test. By default, the tests run against an in-memory H2Database instance, but if you look at application.properties you'll see configurations for PostgreSQL and MySQL as well.

Happy testing!

Posted by Christian Nelson at 3:06 PM | Comments (0)

April 25, 2008

Configuring applications with Spring

If you've used Spring before, you've almost definitely used a PropertyPlaceholderConfigurer to inject settings from external sources -- most likely properties files -- into your application context. The most common use cases include JDBC and Hibernate settings, but it's not that uncommon to also configure Lucene index, temp file, or image cache directories as well. The simplest case looks something like this:

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="location" value="classpath:application.properties"/>
</bean>

<!-- A sample bean that needs some settings. -->
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="${jdbc.driver}"/>
    <property name="url" value="${jdbc.url}"/>
    <property name="username" value="${jdbc.username}"/>
    <property name="password" value="${jdbc.password}"/>
</bean>

And application.properties might look like this:

jdbc.driver=org.h2.Driver
jdbc.url=jdbc:h2:mem:example
jdbc.username=sa
jdbc.password=

Note, you can achieve the same simple configuration using the new spring 2.x style schema configuration, but it doesn't allow for any further customization so we're going to use the old style.

<!-- Example of new Spring 2.x style -->
<context:property-placeholder location="classpath:application.properties"/>

This handles the simple case of replacing placeholders (e.g. ${jdbc.url}) with values found in a properties files (e.g. jdbc.url=jdbc:h2:mem:example). In a real-world application, we not only need to collect settings, but also override them in different environments. Many of our applications are deployed in 4 or more environments (developer machine, build server, staging server, and production), each requiring different databases at the very least.

There are a few ways to enable overriding of properties. Let's take a look at them in turn:

1. Setting the system properties mode to override (default is fallback)

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    <property name="location" value="classpath:application.properties"/>
</bean>

When configured in this mode, any value specified as a system property to the JVM will override any values set in properties files. For example, adding -Djdbc.url=jdbc:h2:mem:cheesewhiz to the JVM arguments would override the value in the file (jdbc:h2:mem:example). On a Java 1.5 or newer platform, Spring will also look for an environment variable called jdbc.url is no system property was found.

2. Specifying an optional properties file

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="ignoreResourceNotFound" value="true"/>
    <property name="locations">
        <list>
            <value>classpath:application.properties</value>
            <value>classpath:local.properties</value>
        </list>
    </property>
</bean>

When ignoreResourceNotFound is set to true, Spring will ignore resources that don't exist. You can imagine application.properties, containing all of the default settings, versioned in your SCM system. Developers have the option of creating a properties file called local.properties to override any settings that differ in their environment. This file should be unversioned and ignored by your SCM system. This works because properties are loaded in order and replace previous values.

3. Web Application overrides

In a web application environment, Spring also supports specifying values in web.xml as context params or in your application server specific meta-data as servlet attributes. For example, if you're using Tomcat you can specify one or more parameter elements in your context.xml, and Spring will can inject those values into placeholders.

<bean class="org.springframework.web.context.support.ServletContextPropertyPlaceholderConfigurer">
    <property name="location" value="classpath:application.properties"/>
</bean>

The ServletContextPropertyPlaceholderConfigurer conveniently works in non servlet environments by falling back to the behavior of a PropertyPlaceholderConfigurer. This is great when running unit tests.

4. Combining techniques

There's no reason why these techniques can't be combined. Technique #1 is great for overriding a few values while #2 is better for overriding many. #3 just expands the field of view when Spring goes to resolve placeholders. When combined, system properties override those in files. When using technique #3, there are some settings available for adjusting the override behavior (see contextOverride). Test the resolution order when combining to ensure it's behaving as expected.

Optional External Properties

There's another use case that applies to some projects. Often in non-developer environments, system admins want to keep properties for the environment outside of the deployable archive or the application server, and they don't want to deal with keeping those files in a Tomcat context file; they prefer a simple properties file. They also don't want to have to place the file in a hard-coded location (e.g. /var/acmeapp/application.properties) or they may keep configuration for multiple servers in the same network directory, each file names after the server. With a little trickery, it's easy to support an optional external properties file that isn't in a hard-coded location. The location of the file is passed as a single system property to the JVM, for example: -Dconfig=file://var/acmeapp/server1.properties. Here's the configuration to make it happen:

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="ignoreUnresolvablePlaceholders" value="true"/>
</bean>

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="ignoreResourceNotFound" value="true"/>
    <property name="location" value="${config}"/>
</bean>

The first definition enables basic property resolution through system properties (in fallback mode). The second bean loads the resource from the location resolved from the system property -Dconfig. All spring resource urls are supported, making this very flexible.

Putting it all together

Here's a configuration that does more than most people would need, but allows for ultimate flexibility:

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="ignoreUnresolvablePlaceholders" value="true"/>
</bean>

<bean class="org.springframework.web.context.support.ServletContextPropertyPlaceholderConfigurer">
    <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/>
    <property name="searchContextAttributes" value="true"/>
    <property name="contextOverride" value="true"/>
    <property name="ignoreResourceNotFound" value="true"/>
    <property name="locations">
        <list>
            <value>classpath:application.properties</value>
            <value>classpath:local.properties</value>
            <value>${config}</value>
        </list>
    </property>
</bean>

Every placeholder goes through the following resolution process. Once a value is found it's set and the next placeholder is resolved:

1. (optional) Property value specified as a system or environment property; useful for overriding specific placeholders.
   e.g. -Djdbc.host=devdb
   e.g. -Djdbc.username=carbon5
2. (optional) Context parameters located in web.xml or context attributes specified in application server meta-data (e.g. a Tomcat context.xml).
3. (optional) Properties file located by the system/environment variable called "config"; useful for externalizing configuration. All URL types are supported.
   e.g. -Dconfig=c://hmc.properties
4. (optional) Properties file identified by classpath:local.properties; useful for specific developer overrides.
5. (required) Properties file identified by classpath:application.properties, which contains default settings for our application.

Best Practices

• Deploy the same exact artifact (e.g. war, ear, etc) across all environments by externalizing configuration. This may seem daunting, but the emergent benefits are huge in terms of simplicity.
• Only make things that can safely change across environments configurable. Also, only things that need to be configurable should be configurable, it's easy to go overboard.
• Configure the minimal properties search path that meets your requirements.
• When looking for properties files in the project tree, use classpath resources whenever possible. This makes finding those files easy, consistent, and insensitive to the working-dir, which is great when running tests from your IDE and command line.
• Aim for a zero-configuration check-out, build, run-tests cycle for the environment where its happens most: development.

Posted by Christian Nelson at 10:57 AM | Comments (0)

March 27, 2008

Injecting Spring 2.5 beans into Stripes Actions

I've been playing around with Stripes, a light-weight, well-designed simple Java web MVC framework recently. I haven't had the pleasure of working with it on a production application yet, but hope to sometime soon. Meanwhile, I've been tinkering on a pet project.

As you may know, we often use Spring for lifecycle management of our services and dependency injection (among other uses). In a Stripes + Spring application, you can imagine Spring-managed services being used by Stripes Actions (i.e. controllers). Every incoming HTTP request results in a new instance of a Stripes Action, thus the newly created actions must have their dependencies injected for every request.

Stripes ships with a SpringInterceptor that supports annotating fields and methods on your actions with @SpringBean. While this works fine, I was pretty interested in being able to use Spring 2.5's annotations for marking what should be injected on my actions, so I created a Spring25Interceptor (see code below).

The Spring25Interceptor is configured the same way as the out-of-the-box SpringInterceptor. In your web.xml:

...
<filter>
    <display-name>Stripes Filter</display-name>
    <filter-name>StripesFilter</filter-name>
    <filter-class>net.sourceforge.stripes.controller.StripesFilter</filter-class>
    <init-param>
        <param-name>Interceptor.Classes</param-name>
        <param-value>
            net.sourceforge.stripes.integration.spring.Spring25Interceptor
        </param-value>
    </init-param>
    ...
</filter>
...

Since it uses the same annotations that Spring 2.5's supports (@Autowired, @Resource, and @Qualifier), annotating your Stripes Actions is easy and should look pretty familiar (see Juergen's blog for a comprehensive overview). Here are a few hypothetical examples:

public class LandingActionBean extends AbstractActionBean
{
     // Autowire by type (looks for a bean in the application context of type ServiceA)
     @Autowired ServiceA serviceA;
 
     // Autowire by name (looks for a bean with the name 'serviceB')
     @Resource ServiceB serviceB;
     
     // Autowire by name (looks for a bean with the name 'serviceC')
     @Autowired @Qualifier("serviceC") ServiceC serviceCee;
     
     // Method injection examples (all of the above can be applied to methods as well)
 
     @Autowired
     public void setServiceA(ServiceA a) { this.serviceA = a; }
     
     @Autowired
     public void setServices(ServiceA a, ServiceB b, ServiceC c) {
          this.serviceA = a;
          this.serviceB = b;
          this.serviceC = c;
      }
 
     @DefaultHandler
     public Resolution execute()
     {
          return new ForwardResolution("/landing.jsp");
      }
}

While it may be considered a subtle improvement, I really like the fact that with this new interceptor, my application has a consistent syntax for dependency injection across the tiers. Additionally, the semantics of the Spring annotations are also consistent and shared throughout. An added bonus is that the new Interceptor is significantly smaller than the existing one.

Here's the Interceptor code that does the dependency injection into actions:

import net.sourceforge.stripes.action.*;
import net.sourceforge.stripes.controller.*;
import net.sourceforge.stripes.util.*;
import org.springframework.beans.factory.config.*;
import org.springframework.context.*;
import org.springframework.util.*;
import org.springframework.web.context.support.*;
import javax.servlet.*;

@Intercepts(LifecycleStage.ActionBeanResolution)
public class Spring25Interceptor implements Interceptor
{
     private static final Log log = Log.getInstance(Spring25Interceptor.class);
 
     public Resolution intercept(ExecutionContext context) throws Exception
     {
          Resolution resolution = context.proceed();
          log.debug("Running Spring dependency injection for instance of ", context.getActionBean().getClass().getSimpleName());
          ServletContext servletContext = StripesFilter.getConfiguration().getServletContext();
          ApplicationContext applicationContext = WebApplicationContextUtils.getWebApplicationContext(servletContext);
          AutowireCapableBeanFactory beanFactory = applicationContext.getAutowireCapableBeanFactory();
          beanFactory.autowireBeanProperties(context.getActionBean(), AutowireCapableBeanFactory.AUTOWIRE_NO, false);
          beanFactory.initializeBean(context.getActionBean(), StringUtils.uncapitalize(context.getActionBean().getClass().getSimpleName()));
          return resolution;
      }
}

I've created a enhancement request for this feature, though the comment-thread is a little bit all over the place. You can find the latest version of the code and Javadoc as an attachment on the issue.

Posted by Christian Nelson at 9:52 AM | Comments (0)

February 6, 2008

Introducing Java DB Migrations

Here at Carbon Five we have the luxury of working on many projects, so anything we can do to make things easier will pay off in multiplicity across new projects. One of the things that we have to deal with on every project is maintaining a database schema over time. We’ve had a manual process of capturing changes in incremental db patch scripts for a while, but it was error prone and sometimes neglected. We’ve been doing more Ruby on Rails work and found Rails Migrations easy to work with and a real time saver. We wanted something that would make our lives easier when working on Java projects in the same way Migrations improve Rails development. With that manifest in mind, Alon and I collaborated on a simple Java database migration framework.

During development, it’s a big deal because each engineer has two instances of the database, one for unit tests and another for running the application. We need an easy way to create a new, up-to-date database and update existing databases. Once a project has launched, it’s a big deal because we need a way to migrate a database teeming with important production data to the latest version without losing critical information.

High Level Requirements

• Initiate a migration from the command-line as a Maven plugin
• Programmatically migrate a database during application startup
• Convention over Configuration
• Initially support migrations written in SQL

At a high level, the migration process looks like this:

1. Query the database (table db_version) to find the current version.
2. Determine the latest database schema version available.
3. If the database is out of date, run each migration in order in its own transaction, updating the db_version for each migration.

We’d identified two usage patterns, the first is more akin to the Rails Migration model in that you explicitly migrate the database via the command line. The second is automatic migration when an application starts up, before Hibernate initializes or any other data access takes place. I’ll discuss each in turn.

Migrating using Maven

This functionality is easy to enable in a mavenized project. First you add the Carbon Five public plugin repository:

<pluginRepositories>
  <pluginRepository>
    <id>c5-public-repository</id>
    <url>http://mvn.carbonfive.com/public</url>
  </pluginRepository>
</pluginRepositories>

And then you configure the migration plugin:

<plugin>
  <groupId>com.carbonfive</groupId>
  <artifactId>maven-migration-plugin</artifactId>
  <version>0.9-SNAPSHOT</version>
  <configuration>
    <defaultEnvironment>test</defaultEnvironment>
    <environments>
      <environment>
        <name>default</name>
        <driver>com.mysql.jdbc.Driver</driver>
        <username>dev</username>
        <password>dev</password>
      </environment>
      <environment>
        <name>test</name>
        <url>jdbc:mysql://localhost/myapp_test</url>
      </environment>
      <environment>
        <name>development</name>
        <url>jdbc:mysql://localhost/myapp_development</url>
      </environment>
    </environments>
  </configuration>
  <dependencies>
    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
      <version>5.1.5</version>
    </dependency>
  </dependencies>
</plugin>

You’ll notice that we’ve got 2 environments configured. You can have as many as you need and you can specify which you want to migrate on the command line. If none are specified the default environment will be migrated. In this example we’re specifying the dependency on our JDBC driver so that the plugin has access to the code it needs to connect the database.

Lastly, you drop in your migration scripts into the src/main/resources/db/migrations directory, naming them using the pattern NNN_description.sql, where NNN is three digits indicating the script sequence. Some examples might be:

• 001_create_users_table.sql
• 002_add_default_users.sql
• 003_add_lastvisit_column.sql

The description is optional and isn’t used for anything, it’s just there so that other developers can get an idea of what a script does without having to open it.

From the command line, you can run the migration plugin like this:

Note that he database must exist for the migrations to take place as we do not create missing databases (yet).

I’ve created a simple, complete sample that shows off this functionality, it’s on the C5 public subversion repository here. Check it out and then read the readme.txt at the top of the project.

Migrating from your Application

The other usage scenario is to auto-migrate during application startup. At the core of the framework, there’s an interface called MigrationManager which has two implementations: DataSourceMigrationManager and DriverManagerMigrationManager. Migration happens right after a datasource (of the javax.sql variety) is created.

Migrating from your application is as easy as instantiating one of these early in the startup cycle and invoking the migrate() method, something like this:

MigrationManager migrationManager = new
    DriverManagerMigrationManager(“com.mysql.jdbc.Driver”, “jdbc:mysql://localhost/myapp_test”,
“dev”, “dev”);
migrationManager.migrate();

Of course this needs to happen before anything else in the application uses the database; we want to database to be updated completely before it’s used.

Spring is part of our standard development stack on our Java projects, and it’s easy to enforce these dependencies in Spring configuration. First we define a data source for the application:

<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
  <property name="driverClassName" value="org.h2.Driver"/>
  <property name="url" value="jdbc:h2:file:~/.h2/migration_sample2_test"/>
  <property name="username" value="dev"/>
  <property name="password" value="dev"/>
</bean>

And now we declare our MigrationManager instance. Note the ‘init-method’ attribute:

<bean id="migrationManager"
    class="com.carbonfive.db.migration.DataSourceMigrationManager"
    init-method="migrate">
  <constructor-arg ref="dataSource"/>
</bean>

And then you define something that’s going to use the defined DataSource. Note the ‘depends-on’ attribute:

 <bean id="userService"
    class="com.carbonfive.migration.sample2.UserService"
    depends-on="migrationManager">
  <constructor-arg ref="dataSource"/>
</bean>

This is obviously a little contrived for the sake of example, but you get the point. In a typical application the thing that would depend on the datasource is a Hibernate SessionFactory.

You can visit the source code for this example on the C5 public subversion repository here.

Best Practices

Here are a few of the things we’ve learned along the way:

• Start using migrations early. Definitely start by time there’s more than one person on a project. I usually start off letting hibernate generate my schema while I’m experimenting with things, but as soon as I’m really working on features I’ll switch over to migrations.
• All database changes are captured as a new migration.
• Migration scripts cannot be changed once *anyone* has run them and make further changes in a new migrations.

Source Code Access

The Carbon Five db-support project which contains all of this migration goodness is available on the C5 public subversion repository at https://svn.carbonfive.com/public/carbonfive/db-support/trunk. It’s a maven project and should compile and pass its tests out of the box. I encourage you to look through the code and check out the tests.

Future

If you look through the code you’ll see some of what’s in store for this project. We’ve got initial support for writing migrations in Groovy and JRuby and we’re thinking about added Java support as well. We’re looking for feedback to drive the future direction of the project, so feel free to write us and let us know what you think.

Posted by Christian Nelson at 4:42 PM | Comments (0)

January 4, 2008

JRubyGems Release

I'm releasing a first version of JRubyGems. This post is documentation until I come up with something better.

JRubyGems allows you to package RubyGems on your Java application classpath. It removes the file system dependencies on locally installed gems that get awkward when using JRuby and dependent gems in a Java environment.

Use JRubyGems as a replacement for RubyGems. Instead of

require 'rubygems'
gem 'activerecord'

use

require 'jrubygems'
gem 'activerecord'

Behind the scenes JRubyGems requires RubyGems and injects behavior specific to finding gems on the Java classpath if they are not already installed.

You make JRubyGems available to your application by putting the jar on your application classpath.

Source is available from Subversion at https://svn.carbonfive.com/public/carbonfive/jruby/jrubygems/trunk/. The tests in ruby/test/test_jrubygems.rb illustrate JRubyGems' behavior.

The jar at http://mvn.carbonfive.com/public/com/carbonfive/jruby/jrubygems/0.3/jrubygems-0.3.jar is all you need to try it out in your own scripts. With JRuby you can require jar files that are in your load path to get them on your classpath and make Ruby files in the jar available on your load path. So if I have a folder lib/ in my load path with the file lib/jrubygems-0.3.jar, I can use JRubyGems directly from a JRuby script with:

require 'jrubygems-0.3.jar'
require 'jrubygems'

Most Java applications using embedded Ruby will have their own classpath management strategy. Just make sure the JRubyGems jar gets on the classpath of your application. If you are using Maven, you can use JRubyGems from the Carbon Five repository:

<repositories>
  <repository>
    <id>c5-public-repository</id>
    <name>Carbon Five Public Repository</name>
    <url>http://mvn.carbonfive.com/public</url>
  </repository>
</repositories>

<dependencies>
  <dependency>
    <groupId>com.carbonfive.jruby</groupId>
    <artifactId>jrubygems</artifactId>
    <version>0.3</version>
  </dependency>
</dependencies>

If you are using Maven you can also require your JRuby dependency with:

<dependency>
  <groupId>org.jruby</groupId>
  <artifactId>jruby-complete</artifactId>
  <version>1.1RC1</version>
</dependency>

Packaging Gem Dependencies

JRubyGems expects a gem to be available on your application classpath as a .gem file (the distribution package for a gem) under the classpath location /gems. This means you can have a gems/ folder on your classpath with a bunch of .gem files in it, a jar with all your .gem files under /gems or a jar per .gem file with each file under /gems in each jar. (A limitation in the JDK's ClassLoader.getResources method which only returns file system locations for a passed-in empty String prevented me from allowing .gem files in jar roots.)

For our migration application, I built a jar for each gem dependency and deployed it to our central Maven repository. My Maven build references these dependencies in my project POM just as I would a jar dependency.

<dependency>
  <groupId>com.carbonfive.jruby.gems</groupId>
  <artifactId>activerecord</artifactId>
  <version>1.15.6</version>
</dependency>

I just grabbed the .gem files from my local JRuby installation's gem cache dir at jruby-1.1b1/lib/ruby/gems/1.8/cache/ to build these jars for now.

I like that Raven takes exactly the opposite approach - managing jar dependencies as gems.

Christian and I have a side project at Carbon Five to create a standard mechanism for managing database migrations for our Java projects. The first implementation defines a migration as a SQL script to be run against the target database. I thought it would be cool to support ActiveRecord migrations too, especially for providing an easy way to do the data migrations that accompany a schema change.

My prototype of this idea went well (more on that some other time) but I quickly ran into an issue with the gem dependency for 'activerecord-jdbc-adapter' and its dependencies 'activesupport' and 'activerecord'. The issue is that my Java project was nicely portable but the gems create a dependency on the runtime system having specific gems installed. If I install the gems locally, I also have to install them on other runtime systems like our continuous integration server. I found another example of a JRuby user running into this problem when reading about this example of using the RedCloth Ruby library to format text in a Spring/Java application. Note the "ugly underbelly" footnote.

In the Ruby world, this is the way things work. Large Ruby applications (especially Rails applications) often rely on system services like cron to do their work. These applications have their own mechanisms for setting up new runtime systems. In the Java world, we expect to package our application with all of its dependencies and deploy it to little more than a JVM and webapp container.

Java has the classpath abstraction to address minimizing file system dependencies while Ruby uses the file system directly and extensively. I've seen a couple approaches for addressing this mismatch between Java and Ruby. All of them involve taking some set of resources packaged in a Java archive (jar, war), extracting it to a file system location at runtime and configuring Ruby to use the resources at that location. The "jruby-complete" JRuby distribution unpacks the core Ruby libraries into ~/.jruby/. GoldSpike, the Rails plugin for packaging Rails applications as war files, bundles gems in WEB-INF and configures GEM_HOME at runtime to use the gems from the unpacked war.

I decided to take a similar approach with JRubyGems - bundle gem dependencies in the application classpath and install them on the local file system on demand if they are not already available. As I expect happens with many JRuby projects, it took a little Java and a little Ruby and works quite well.

The main flow for RubyGems to load a gem that has not yet been loaded for an application is:

1. Kernel::gem
2. Kernel::activate_gem_with_options
3. Gem.activate
4. Gem.source_index.find_name
5. Gem.activate each dependency of this gem first
6. Finish activating this gem

JRubyGems replaces the Gem.activate implementation to insert a couple steps that:

1. Check if the gem is locally installed
2. Search the Java classpath for the gem source if not installed
3. Install the gem locally
4. Continue with the base Gem.activate implementation

This ensures that gems and their transitive dependencies are installed and loaded.

The classpath searching is implemented in Java.

Release History

0.3 2008-01-28

• Upgrade to use JRuby 1.1RC1 and RubyGems 1.0.1 (bundled with 1.1RC1)
• Added test for gem that bundles a jar in lib/ (hpricot)

0.2 2008-01-07

• Resolved issues with transitive dependency installation
• Force install of classpath gems to avoid dependency errors
• Additional logging of install locations

0.1 2008-01-04

Initial release.

Posted by Alon Salant at 12:39 PM | Comments (1)

January 2, 2008

Thoughts on JRuby

At Carbon Five we have built our professional consulting practice on the solid foundation of enterprise Java development. In 2007 we added Ruby and Ruby on Rails as development tool and framework that complement our existing values and process in many fantastic ways.

Looking forward, I want to be sure that we take maximum advantage of our existing knowledge and new learning in both Java and Ruby. I have high hopes for JRuby as a valuable bridge that spans both languages and gives developers the flexibility to use the strengths of the two. Sun appears to feel the same way as they have hired the two core JRuby developers, Charles Nutter and Thomas Enebo, to work on JRuby full time.

In preparation for more specific posts, here are some light musings on JRuby.

JRuby 1.0

Released mid-2007 JRuby 1.0 was focused on Ruby compatibility. The idea is that any Ruby script or application you can run with the native C Ruby interpreter, you can run with JRuby. The success of this effort can be observed with Ruby on Rails applications now running on JRuby. Of course, there are some limitations.

In addition to running pure Ruby, JRuby can access the Java environment in which it is running. You get the Ruby niceties that you would expect like optional conversion of CamelCase method names to underscore_separated.

list = java.util.ArrayList.new
list.add("one")
list.add_all [2, "three"]

Any Java classes in the Java classpath are available to your JRuby script. Additionally, you can 'require foo.jar' to make it available on your classpath at runtime.

You can also extend Java classes in JRuby.

class MyList < java.util.ArrayList
  def say_hi
    "hi"
  end
end

Internally, many core Ruby classes have been rewritten to delegate to core Java classes.

JRuby 1.1

JRuby 1.1 is presently in beta. The top priorities for JRuby 1.1 are performance and Java integration. Ruby is not fast. JRuby 1.0 was worse. Performance was not a priority for 1.0 and the consensus is that there are a lot of easy performance wins that it is now time to take advantage of. In his blog, Charles Nutter states:

Straight-line execution is now typically 2-4x better than Ruby 1.8.6. Rails whole-app performance ranges from just slightly slower to slightly better, though there have been reports of specific apps running many times faster.

Java Integration

Java integration refers to features for the using Ruby from Java rather than Java from Ruby as I discuss above. When I first started looking at JRuby Christian and I were working on a Java library to manage database migrations ala RoR and some other neat tricks we've seen along the way. (More on that later.) I thought it would be cool if you could write migrations using ActiveRecord's migration features and invoke them from Java. I found myself looking for the easy means to invoke Ruby from Java. Some of this is in place with more slated for future JRuby versions.

The Spring Framework provides JRuby integration through which you can expose Ruby classes as Spring-managed beans. This example of using the RedCloth Ruby library to format text in a Java application is a pretty slick example of how this works.

This post does point out one issue that I ran into with using Ruby from Java - RubyGems. Gems are Ruby libraries available from central repositories and installed on the system running Ruby. For Java applications, this dependency on system-installed resources is lame. To address this issue, I've spent some time working on a solution for packaging Ruby gem dependencies in Java applications. More on JRubyGems coming soon.

Posted by Alon Salant at 2:00 PM | Comments (0)

June 11, 2007

Parameterized REST URLs with Spring MVC

At Carbon Five, we've been working REST-ful practices into our web applications for some time now. Providing simple URLs for application entities is a key principal of this style, but parsing parameters out of the request path has been klunky in Spring MVC. Spring's WebFlow apparently supports REST-ful URLs, but I've never found anything in what I've read or heard to recommended that project (though I've heard nothing bad).

I finally got fed up with the situation and worked out a solution that lets developers specify path parameters in the dispatcher mappings, which will appear as request parameters in the controller and view. I've made the project available on our public SVN server. The solution requires only 4 classes, so you can download them instead. Read on for configuration information.

 <bean class="carbonfive.spring.web.pathparameter.ParameterizedUrlHandlerMapping">
   <property name="alwaysUseFullPath" value="true"/>
   <property name="mappings">
     <props>
       <prop key="/view/noparameters">controller1</prop>
       <prop key="/view/(bar:foo)">controller2</prop>
       <prop key="/view/(*.html:html)">controller3</prop>
       <prop key="/view/(**/*:view).view">controller4</prop>
       <prop key="/view/c/(*:controller)/(*:id)">controller5</prop>
     </props>
   </property>
 </bean>

  <filter>
    <filter-name>PathParameterFilter</filter-name>
    <filter-class>carbonfive.spring.web.pathparameter.ParameterizedPathFilter</filter-class>
  </filter>

  <filter-mapping>
    <filter-name>PathParameterFilter</filter-name>
    <url-pattern>/path/to/dispatcher/servlet/*</url-pattern>
  </filter-mapping>

Posted by Alex Cruikshank at 9:45 AM | Comments (19)

June 8, 2007

Using UrlRewriteFilter with the Spring Framework

We typically register one Spring FilterToBeanProxy and configure our filters using FilterChainProxy. (Note: these classes referenced are actually part of the ACEGI security framework for Spring.) This keeps the web.xml simple and gives us all the power of Spring for configuring and initializing our filters. In this model, you typically use setter-based injection to configure your filter. In the case of UrlRewriteFilter, the filter must be initialized with the Filter API method init and a FilterConfig object. The problem I needed to solve was how to create and initialize this filter, or any servlet filter really, from Spring where init must be called with initialization parameters in the FilterConfig.

My solution is a fairly standard approach that you use when integrating 3rd party libraries with Spring - create a FactoryBean. The Spring FactoryBean interface provides a simple mechanism for creating objects with life cycle constraints that make them hard to create using Spring's standard mechanisms.

Here's my implementation of a FilterFactoryBean. Note that this FactoryBean could be used to initialize any servlet filter in Spring.

package carbonfive.springframework.filter;

import javax.servlet.*;
import java.util.*;

import org.springframework.beans.factory.*;
import org.springframework.web.context.*;
import org.apache.commons.logging.*;

/**
 * FactoryBean implementation for creating and initializing a servlet filter for use
 * in a Spring context. Written to handle initializing UrlRewriteFilter.
 */
public class FilterFactoryBean implements FactoryBean, ServletContextAware, InitializingBean
{
   Log log = LogFactory.getLog(FilterFactoryBean.class);
 
   private Hashtable initParameters;
   private Class filterClass;
   private ServletContext servletContext;
 
   public Object getObject() throws Exception
   {
      log.info("Creating filter " + filterClass.getName());
      
      Filter filter = (Filter) filterClass.newInstance();
      FilterConfig config = new FilterConfigImpl(initParameters);
      filter.init(config);
      return filter;
    }
 
   public Class getObjectType()
   {
      return Filter.class;
    }
 
   public boolean isSingleton()
   {
      return true;
    }
 
   public void setFilterClass(Class filterClass)
   {
      this.filterClass = filterClass;
    }
 
   public void setInitParameters(Hashtable initParameters)
   {
      this.initParameters = initParameters;
    }
 
   public void setServletContext(ServletContext servletContext)
   {
      this.servletContext = servletContext;
    }
 
   public void afterPropertiesSet() throws Exception
   {
      if (filterClass == null) throw new IllegalArgumentException("filterClass can not be null");
      if (!Filter.class.isAssignableFrom(filterClass)) throw new IllegalArgumentException("filterClass must implement javax.servlet.Filter");
    }
 
   private class FilterConfigImpl implements FilterConfig
   {
      Hashtable properties;
  
      public FilterConfigImpl(Hashtable properties)
      {
         this.properties = properties;
       }
  
      public String getFilterName()
      {
         return filterClass.getName() + "-filter";
       }
  
      public ServletContext getServletContext()
      {
         return servletContext;
       }
  
      public String getInitParameter(String name)
      {
         return (String) properties.get(name);
       }
  
      public Enumeration getInitParameterNames()
      {
         return properties.keys();
       }
    }
}

And a test for this implementation using UrlRewriteFilter:

package carbonfive.springframework.filter;

import javax.servlet.*;
import java.util.*;

import junit.framework.*;
import org.tuckey.web.filters.urlrewrite.*;
import org.apache.commons.logging.*;
import org.springframework.mock.web.*;

public class FilterFactoryBeanTest extends TestCase
{
   Log log = LogFactory.getLog(FilterFactoryBeanTest.class);
 
   public void testCreateUrlRewriteFilter() throws Exception
   {
      log.info("Starting test");
      
      FilterFactoryBean factory = new FilterFactoryBean();
      factory.setServletContext(new MockServletContext());
      factory.setFilterClass(UrlRewriteFilter.class);
  
      Properties properties = new Properties();
      properties.put("confPath", "/WEB-INF/rewrite.xml");
      properties.put("confReloadCheckInterval", "30");
      properties.put("logLevel", "DEBUG");
      properties.put("statusPath", "/status");
      properties.put("statusEnabled", "true");
      properties.put("statusEnabledOnHosts", "localhost,127.0.0.1");
  
      factory.setInitParameters(properties);
      factory.afterPropertiesSet();
      
      assertEquals(Filter.class, factory.getObjectType());
  
      Filter filter = (Filter) factory.getObject();
      assertNotNull(filter);
      assertTrue(filter instanceof UrlRewriteFilter);
  
      UrlRewriteFilter rewriter = (UrlRewriteFilter) filter;
      assertEquals(30, rewriter.getConfReloadCheckInterval());
      assertEquals(true, rewriter.isConfReloadCheckEnabled());
      assertEquals("/status", rewriter.getStatusPath());
      assertEquals(true, rewriter.isStatusEnabled());
    }
}

To create this filter in Spring my configuration looks like:

  <bean id="smilemaker-urlRewriteFilter" class="carbonfive.springframework.filter.FilterFactoryBean">
    <property name="filterClass" value="org.tuckey.web.filters.urlrewrite.UrlRewriteFilter"/>
    <property name="initParameters">
      <props>
        <prop key="confPath">/WEB-INF/config/urlrewrite.xml</prop>
        <prop key="confReloadCheckInterval">10</prop>
        <prop key="statusEnabled">true</prop>
        <prop key="statusPath">/admin/rewrite-status</prop>
      </props>
    </property>
  </bean>

Note that I found and reported (and locally fixed) a bug in UrlRewriteFilter where changes to a config file in a non-standard location do not get picked up.

Posted by Alon Salant at 11:52 AM | Comments (0)

May 17, 2007

Using classpath*: vs. classpath: when loading Spring resources.

I recently resolved a build problem nestled deep within the esoterica of spring resource loading. The behavior of classpath: URLs is explained at length in the Spring documentation, but, sadly, this is probably the last place a developer of my disposition would look. The problem (which I describe in more detail beyond the fold) was that we were unable to load hibernate mapping files from our Spring configured integration tests even though they gave us no trouble in our web application. The solution was to replace the "classpath:" prefix of our mappingLocation URI with "classpath*:".

In this project, we are storing our Hibernate mapping files as separate files located in the package of the class they map (i.e. "Foo.hbm.xml" stored in "src/main/java/com/example/"). We inherited the persisted classes and Hibernate and Spring configuration from another project. The sessionFactory configuration in spring specified the mappingLocations as "classpath:**/*.hbm.xml" which worked fine when we deployed the web application.

We soon ran into problems when we added integration tests to test new DAO functionality. The test configuration was not able to find any hibernate mappings even though we were using the same "classpath:**/*.hbm.xml" URI. After a few hours of hair-pulling and a deep tour of the Spring source, we realized that the problem was that our test Spring configuration was in a different classpath root from the Hibernate mappings, even though both directories were in the test's classpath.

When you use 'classpath:' for an Ant style wildcard search, Spring uses a single classpath directory for the search. The documentation is vague, but it seems the directory returned will be the first one provided by ClassLoader.getResources(""). In our case, it returned the '/target/test-classes' directory that contains applicationTest.xml and our test classes, instead of '/target/classes' which contains application.xml and all the *.hbm.xml files.

Using the 'classpath*:' prefix fixes the problem. It indicates that the resource loader should look in all directories on the classpath, so making this change solved our problem. Apparently Spring maintains both prefixes because limitations in the Classloader (at the specification level) make it difficult to search for resources in the classpath root when performing wildcard searches across all classpath directories. This suggest it might be good practice to always create a directory to contain resources you might otherwise want to put in the classpath root and always use the 'claspath*:' prefix.

Posted by Alex Cruikshank at 10:59 AM | Comments (0)

May 15, 2007

DBUnit 2.2, Spring, and Testing

You may have noticed that DBUnit changed its connection closing behavior in v2.2. We noticed it when tests deriving from our custom DatabaseTestCase implementation (which uses DBUnit) starting failing. Initially we just reverted back to version 2.1. Since then I've discovered what the problem was and found a workaround, and while I was at it, I created some nice utilities to be used in tests which need data fixtures.

Problem: In v2.2 DBUnit introduces some new abstractions, one of which is the IDatabaseTester. Whether it’s by design or by accident, the default implementation closes connections after executing its operations (more info). The end result is that in our unit tests, the database connection is closed before the test can run.

Solution: I’ve created two simple util classes for loading test data into a database without closing the connection:

/** A helper for loading data sets into unit tests. */
public class DatabaseUtils {
     public static void loadDataSet(Class clazz, final DataSource dataSource) throws Exception {
          IDataSet dataSet = new FlatXmlDataSet(TestUtils.datasetInputStream(clazz));
          IDatabaseTester tester = new ExistingConnectionDatabaseTester(dataSource);
          tester.setDataSet(dataSet);
          tester.onSetup();
      }
}

 
/** A special DatabaseTester that doesn’t close the connection when its done. */
public class ExistingConnectionDatabaseTester extends AbstractDatabaseTester {
     private DataSource dataSource;
  
     public ExistingConnectionDatabaseTester(DataSource dataSource) {
          super();
          this.dataSource = dataSource;
      }
  
     public IDatabaseConnection getConnection() throws Exception {
          return new DatabaseConnection(DataSourceUtils.getConnection(dataSource));
      }
  
     public void closeConnection(IDatabaseConnection connection) throws Exception {
          // Don't close that connection!
      }
}

The first of these depends on another test-related utility, TestUtils.java (see below), which converts a class to a path. The second depends on Spring’s DataSourceUtils.

 
/**
 * A set of utilities that generate paths from classnames.  This is useful when
 * making reference to resources used by test cases, when the resources are
 * located in a directory path matching the class' package.
 */
public class TestUtils {
 
     public static final String TEST_PREFIX = "src/test/resources/"; // Maven2 default
 
     public static String pathString(Class clazz) {
          return pathString(TEST_PREFIX, clazz, "");
      }
 
     public static String pathString(Class clazz, String resource) {
          return pathString(TEST_PREFIX, clazz, resource);
      }
 
     public static String pathString(String prefix, Class clazz, String resource) {
          prefix = (prefix != null ? prefix : "");
  
          StringBuffer sb = new StringBuffer();
          sb.append(prefix);
  
          if (!prefix.endsWith("/")) {
               sb.append("/");
           }
  
          sb.append(ClassUtils.classPackageAsResourcePath(clazz));
          sb.append("/");
          sb.append(resource);
  
          return sb.toString();
      }
 
     public static InputStream pathInputStream(Class clazz, String resource) throws FileNotFoundException {
          return pathInputStream(TEST_PREFIX, clazz, resource);
      }
 
     public static InputStream pathInputStream(String prefix, Class clazz, String resource) throws FileNotFoundException {
          return new FileInputStream(pathString(prefix, clazz, resource));
      }
 
     public static InputStream datasetInputStream(Class clazz) throws FileNotFoundException {
          return pathInputStream(TEST_PREFIX, clazz, ClassUtils.getShortName(clazz) + ".xml");
      }
}

These three classes give us everything we need to load data into our test database using DBUnit. And TestUtils can be used to load other test resources from the classpath as well, like images, documents, CSVs, etc.

Here's an example of how it can be used in a DAO test based on Spring's test classes.

public class ArtistHibernateDaoTest extends AbstractAnnotationAwareTransactionalTests {
  
     private ArtistHibernateDao artistDao;
  
     public void setArtistDao(ArtistHibernateDao artistDao) {
          this.artistDao = artistDao;
      }
  
     protected String[] getConfigLocations() {
          return new String[]{"applicationContext-database.xml", "applicationContext-hibernate.xml"};
      }
  
     protected void onSetUpInTransaction() throws Exception {
          DatabaseUtils.loadDataSet(getClass(), getJdbcTemplate().getDataSource());
      }
  
     public void testFindAllNames() {
          List<String> names = artistDao.findAllNames();
          assertEquals(3, names.size());
          assertTrue(names.contains("The Cure"));
          assertTrue(names.contains("Depeche Mode"));
          assertTrue(names.contains("New Order"));
      }
}

onSetUpInTransaction() loads the data set using the convention of (packagename).(ClassName).xml before each test (e.g. com.c5.PizzaTest loads the file com/c5/PizzaText.xml on the classpath). In this case the data is loaded in the same transaction as the test so that no clean-up is necessary when the test is completed. These tests run fast!

We've been talking about moving away from our custom DatabaseTestCase class hierarchy. The above functionality in conjunction with Spring's test hierarchy could be a good start.

On a somewhat related note, there is a new unit testing framework called Unitils which does this sort of thing and much more. I haven’t used it but it sounds interesting.

Posted by Christian Nelson at 9:30 AM | Comments (0)

April 3, 2006

Presenting Spring AOP at SDForum April 4

I will be discussing a model for using the AOP features of Spring to expose Hibernate-managed POJOs to remote clients through web services complete with role-based security and declarative transactions. It follows one of the service API design approaches I discussed in this forum last year.

The source for this presentation is available here.

Posted by Alon Salant at 3:19 PM | Comments (4)

September 12, 2005

SsTemplates Spreadsheet Templates for Excel

SsTemplates builds upon the Jakarta POI/HSSF libraries for writing Microsoft Office documents by providing a simple XML templating language for creating Excel documents similar to creating HTML pages with JSP and CSS.

An example from the documentation:

<workbook xmlns="http://carbonfive.com/schema/sstemplates">
  <style name="label" fontWeight="bold"/>
  <style name="money" dataFormat="$0.00"/>

  <sheet name="Formula Spreadsheet">
    <row>
      <cell style="label">January</cell>
      <cell style="money" type="numeric">1.99</cell>
    </row>
    <row>
      <cell style="label">February</cell>
      <cell style="money" type="numeric">2.09</cell>
    </row>
    <row>
      <cell style="label">March</cell>
      <cell style="money" type="numeric">3</cell>
    </row>
    <row>
      <cell style="label">Total</cell>
      <cell style="money" type="formula">sum(B1:B3)</cell>
    </row>
  </sheet>
</workbook>

Posted by Alon Salant at 10:53 AM | Comments (0)