Integration and Functional Testing with Maven 2.0

Intro

"The Failsafe Plugin is designed to run integration tests while the Surefire Plugins is designed to run unit tests." - http://maven.apache.org/plugins/maven-failsafe-plugin/

See below for the Failsafe section.

However, some people are using Surefire for integration testing - not least because Failsafe only came along some time after Surefire ...

Put your Integration tests in a separate module

Better Builds With Maven describes a single example module containing only integration tests. In the case of a multi-module "reactor" project, where each module may need some integration testing, you have some choices available to you.

Create Multiple Integration Test POMs

You can, if you wish, create multiple integration test modules, one for each source module in your project. For example, if you've got two child projects "foo" and "bar", you could create a "foo-integration-test" module and a "bar-integration-test" module to test "foo" and "bar" respectively.

The advantage of doing it this way is that it's very clear which integration tests belong to which project. It's also easy for project owners to know which integration tests they're responsible for. The big drawback here is that you've just doubled the number of POM files in your reactor, which can be difficult to manage. Each project will need to be separately referenced in the root POM.

Put all Integration Tests into One Big Module

This is obviously much simpler to wire up in a POM file; simplicity is a virtue in project management.

The disadvantage of doing it this way is that it tends to separate the integration tests from the code they're attempting to test. As a result, you may find that no one "owns" the integration tests; typically you'll have some one person whose job it is to analyze the integration tests and find bugs. QA is hard, but it's even harder when it's unclear who "owns" test failures.

Alternatives to Isolating the Integration Tests

If for some reason you can't put the integration tests in a separate module, here are some ideas.

Integration Tests Only

If you have only integration tests in the same module as your webapp, you can configure Surefire to skip the test phase, then run in the integration-test phase. See this page.

Both Unit and Integration Tests in the Same Module

If you need to run both unit and integration tests in the same module, it's possible, just not very pretty.

There is only one testSourceDirectory per module, so all of your test classes must reside in one directory structure, usually src/test/java.

In the 'test' phase, configure Surefire to exclude the tests in (for example) **/systest/**.

Bind another execution of maven-surefire-plugin to the integration-test phase, and reverse the exclusion pattern.

You can see an example of this in the Shale Usecases example app pom.xml file.

Using the Maven Failsafe Plugin to run Integration Tests

The Maven Failsafe Plugin is a fork of the Maven Surefire Plugin designed to help when running integration tests.

If you use this approach, you keep all your tests for a module in the testSourceDirectory, e.g. src/test/java. By default the Failsafe Maven Plugin looks for integration tests matching the patterns */IT.java, **/IT.java and */*ITCase.java.  You will notice that these bindings do not overlap with the default surefire bindings.  To use the Maven Failsafe Plugin you need to add the following to your pom.xml file.

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.6</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

You will then have the following lifecycle bindings

Phase

Plugin execution goal

test

surefire:test

integration-test

failsafe:integration-test

verify

failsafe:verify

The advantage to using the Maven Failsafe Plugin is that it will not stop the build during the integration-test phase if there are test failures.  The recommendation is that you do not directly invoke the pre-integration-test, integration-test or post-integration-test phases but that instead you run integration tests by specifying the verify phase, e.g.

mvn verify

This allows you to set-up your integration test environment during the pre-integration-test phase, run your integration tests during the integration-test phase, cleanly tear-down your integration test environment during the post-integration-test phase before finally checking the integration test results and failing the build if necessary. Here is an example using jetty for hosting an integration test environment:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.6</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.1.16</version>
        <configuration>
          <scanIntervalSeconds>10</scanIntervalSeconds>
          <stopPort>8005</stopPort>
          <stopKey>STOP</stopKey>
          <contextPath>/</contextPath>
        </configuration>
        <executions>
          <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <scanIntervalSeconds>0</scanIntervalSeconds>
              <daemon>true</daemon>
            </configuration>
          </execution>
          <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

The Future

Rumor has it that a future version of Maven will support something like src/it/java in the integration-test phase, in addition to src/test/java in the test phase.

External Links

Surefire

Failsafe

Related Articles