Maven and Integration Testing

Integration and Functional Testing with Maven 2.0

Put your Integration tests in a separate module

Better Builds With Maven describes a single example module containing only integration tests. In the case of a multi-module "reactor" project, where each module may need some integration testing, you have some choices available to you.

Create Multiple Integration Test POMs

You can, if you wish, create multiple integration test modules, one for each source module in your project. For example, if you've got two child projects "foo" and "bar", you could create a "foo-integration-test" module and a "bar-integration-test" module to test "foo" and "bar" respectively.

The advantage of doing it this way is that it's very clear which integration tests belong to which project. It's also easy for project owners to know which integration tests they're responsible for. The big drawback here is that you've just doubled the number of POM files in your reactor, which can be difficult to manage. Each project will need to be separately referenced in the root POM.

Put all Integration Tests into One Big Module

This is obviously much simpler to wire up in a POM file; simplicity is a virtue in project management.

The disadvantage of doing it this way is that it tends to separate the integration tests from the code they're attempting to test. As a result, you may find that no one "owns" the integration tests; typically you'll have some one person whose job it is to analyze the integration tests and find bugs. QA is hard, but it's even harder when it's unclear who "owns" test failures.

Alternatives to Isolating the Integration Tests

If for some reason you can't put the integration tests in a separate module, here are some ideas.

Integration Tests Only

If you have only integration tests in the same module as your webapp, you can configure Surefire to skip the test phase, then run in the integration-test phase. See this page.

Both Unit and Integration Tests in the Same Module

If you need to run both unit and integration tests in the same module, it's possible, just not very pretty.

There is only one testSourceDirectory per module, so all of your test classes must reside in one directory structure, usually src/test/java.

In the 'test' phase, configure Surefire to exclude the tests in (for example) **/systest/**.

Bind another execution of maven-surefire-plugin to the integration-test phase, and reverse the exclusion pattern.

You can see an example of this in the Shale Usecases example app pom.xml file.

Using the Failsafe Maven Plugin to run Integration Tests

The Failsafe Maven Plugin is a fork of the Maven Surefire Plugin designed to help when running integration tests.

If you use this approach, you keep all your tests for a module in the testSourceDirectory, e.g. src/test/java. By default the Failsafe Maven Plugin looks for integration tests matching the patterns */IT.java, **/IT.java and */*ITCase.java.  You will notice that these bindings do not overlap with the default surefire bindings.  To use the Failsafe Maven Plugin you need to add the following to your pom.xml file.

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>failsafe-maven-plugin</artifactId>
        <version>2.4.3-alpha-1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

You will then have the following lifecycle bindings

Phase
Plugin execution goal
test surefire:test
integration-test failsafe:integration-test
verify failsafe:verify

The advantage to using the Failsafe Maven Plugin is that it will not stop the build during the integration-test phase if there are test failures.  The recommendation is that you do not directly invoke the pre-integration-test, integration-test or post-integration-test phases but that instead you run integration tests by specifying the verify phase, e.g.

mvn verify

This allows you to set-up your integration test environment during the pre-integration-test phase, run your integration tests during the integration-test phase, cleanly tear-down your integration test environment during the post-integration-test phase before finally checking the integration test results and failing the build if necessary. Here is an example using jetty for hosting an integration test environment:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>failsafe-maven-plugin</artifactId>
        <version>2.4.3-alpha-1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.1.16</version>
        <configuration>
          <scanIntervalSeconds>10</scanIntervalSeconds>
          <stopPort>8005</stopPort>
          <stopKey>STOP</stopKey>
          <contextPath>/</contextPath>
        </configuration>
        <executions>
          <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <scanIntervalSeconds>0</scanIntervalSeconds>
              <daemon>true</daemon>
            </configuration>
          </execution>
          <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

The Future

Rumor has it that a future version of Maven will support something like src/it/java in the integration-test phase, in addition to src/test/java in the test phase.

Related Articles

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Comments (4)

 

  1. Mar 20, 2008

    Mike Haller says:

    How to you do reporting (such as coverage with cobertura) on core modules using ...

    How to you do reporting (such as coverage with cobertura) on core modules using a separate integration test module? Afaik there is no way on instrumenting the core module's classes when they're referenced as an already-packaged jar file.

  2. Oct 07, 2008

    Bernard Perchman says:

    An alternative to this which is slightly cleaner is to use profiles. Profiles an...

    An alternative to this which is slightly cleaner is to use profiles. Profiles and inheritance are a little buggy so some workarounds were required; however, it can be done as follows:

    1. Create a property to point to your standard test area. 

     <properties>
             <srcdir>src/test/java</srcdir>
     </properties>

    2. Create a profile with a named id such as integration-test. The profile acts to overwrite the source directory - and can also pull in other resources that are integration specific

     <profiles>
      <profile>
       <id>integration-test</id>
           <properties>
             <srcdir>src/integration/java</srcdir>
           </properties>
      </profile>
     </profiles>

    3. In your build stanza, refer to the source directory using the variable

     <build>
       <testSourceDirectory>$

    Unknown macro: {srcdir}

    </testSourceDirectory>
     </build>

    4.To execute in "integration mode", add a -Pintegration-test to your maven call.

    If you place this in your top-level pom it will automatically filter through to the children. One aspect of this configuration is that your non-integration tests aren't executed.

  3. Jan 09, 2009

    Jean-Claude says:

    When the unit tests and the integration tests are in the same source directory b...

    When the unit tests and the integration tests are in the same source directory but not in the same package, there are 3 solutions.

    Solution 1: The solution you explain in shale-usecases

    Solution 2: A solution suggested by John Fergusson in  John Ferguson Smart's Blog

    Solution 3: A solution suggested by Rod Coffin in Wiki: Maven Integration Testing

    Personally  I prefer the solution 3 because it doesn't use a profile (used in solution 1) and it doesn't skip any configuration (strategy used in solution 2).

    What's your opinion?

    Appendix

    A. The structure of the project

     +---main
|   +---java
|       +---ch
|       |   +---example
|       |           +---utest
|       |               +---simple
|       |                   |   Service.java
|               
\---test
    +---java
        +---ch
        |   +---example
        |           +---utest
        |               +---simple
        |                   |   ServiceTest.java
        \---itest
            +---ch
            |   +---example
            |           +---utest
            |               +---simple
            |                   |   ServiceTest.java

     The unit tests are under ch.example.utest.simple, the integration tests are under itest.ch.example.utest.simple

    B. The corresponding POM concerning the solution 1

     ...
  <build>
    <plugins>

      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <excludes>
            <exclude>**/itest/**</exclude>
          </excludes>
        </configuration>
      </plugin>

    </plugins>
  </build>

...

  <profiles>

    <profile>
      <id>itest</id>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <executions>
              <execution>
                <id>surefire-itest</id>
                <phase>integration-test</phase>
                <goals>
                  <goal>test</goal>
                </goals>
                <configuration>
                  <excludes>
                    <exclude>none</exclude>
                  </excludes>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>

  </profiles>


    C. The corresponding POM concerning the solution 2

     ...
  <build>
    <plugins>

      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <skip>true</skip>
        </configuration>
        <executions>
          <execution>
            <id>surefire-test</id>
            <phase>test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <skip>false</skip>
              <excludes>
                <exclude>**/itest/**</exclude>
              </excludes>
            </configuration>
          </execution>

          <execution>
            <id>surefire-itest</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <skip>false</skip>
              <includes>
                <include>**/itest/**</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>

    </plugins>
  </build>
...


    D. The corresponding POM concerning the solution 3

    ...
  <build>
    <plugins>

      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <excludes>
            <exclude>**/itest/**</exclude>
          </excludes>
        </configuration>
        <executions>

          <execution>
            <id>surefire-itest</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <excludes>
                <exclude>none</exclude>
              </excludes>
              <includes>
                <include>**/itest/**</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>

    </plugins>
  </build>
...
 



  4. Jan 12, 2009

    Pete Butterfill says:

    Another alternative is to configure the maven-surefire-plugin exclude using a sy...

    Another alternative is to configure the maven-surefire-plugin exclude using a system property.

    The POM: 

    <build>
        <plugins>
            ...
            <plugin>
                <!--
                Use system property to separate integration tests from unit tests.

                For goals that do not want to run integration tests (e.g. install or test)
                the system property "surefireExclude" should be set to "**/*IntegrationTest.java".

                For goal that should run integration tests (e.g. site:site or integration-test)
                the system property "surefireExclude" should be not set or set to "none".

                See also NetBeans project, properties, actions.
                -->
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>${surefireExclude}</exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

    then test as:  

    mvn -DsurefireExclude=**/*IntegrationTest.java test

    build as:

    mvn -DsurefireExclude=**/*IntegrationTest.java install

    and run integration test as:

    mvn integration-test


    This example assumes that your integration test class names end with IntegrationTest but you could use a directory name convention instead.