Skip to end of metadata
Go to start of metadata

Integration and Functional Testing with Maven 2.0

Intro

"The Failsafe Plugin is designed to run integration tests while the Surefire Plugins is designed to run unit tests." - http://maven.apache.org/plugins/maven-failsafe-plugin/

See below for the Failsafe section.

However, some people are using Surefire for integration testing - not least because Failsafe only came along some time after Surefire ...

Put your Integration tests in a separate module

Better Builds With Maven describes a single example module containing only integration tests. In the case of a multi-module "reactor" project, where each module may need some integration testing, you have some choices available to you.

Create Multiple Integration Test POMs

You can, if you wish, create multiple integration test modules, one for each source module in your project. For example, if you've got two child projects "foo" and "bar", you could create a "foo-integration-test" module and a "bar-integration-test" module to test "foo" and "bar" respectively.

The advantage of doing it this way is that it's very clear which integration tests belong to which project. It's also easy for project owners to know which integration tests they're responsible for. The big drawback here is that you've just doubled the number of POM files in your reactor, which can be difficult to manage. Each project will need to be separately referenced in the root POM.

Put all Integration Tests into One Big Module

This is obviously much simpler to wire up in a POM file; simplicity is a virtue in project management.

The disadvantage of doing it this way is that it tends to separate the integration tests from the code they're attempting to test. As a result, you may find that no one "owns" the integration tests; typically you'll have some one person whose job it is to analyze the integration tests and find bugs. QA is hard, but it's even harder when it's unclear who "owns" test failures.

Alternatives to Isolating the Integration Tests

If for some reason you can't put the integration tests in a separate module, here are some ideas.

Integration Tests Only

If you have only integration tests in the same module as your webapp, you can configure Surefire to skip the test phase, then run in the integration-test phase. See this page.

Both Unit and Integration Tests in the Same Module

If you need to run both unit and integration tests in the same module, it's possible, just not very pretty.

There is only one testSourceDirectory per module, so all of your test classes must reside in one directory structure, usually src/test/java.

In the 'test' phase, configure Surefire to exclude the tests in (for example) **/systest/**.

Bind another execution of maven-surefire-plugin to the integration-test phase, and reverse the exclusion pattern.

You can see an example of this in the Shale Usecases example app pom.xml file.

Using the Maven Failsafe Plugin to run Integration Tests

The Maven Failsafe Plugin is a fork of the Maven Surefire Plugin designed to help when running integration tests.

If you use this approach, you keep all your tests for a module in the testSourceDirectory, e.g. src/test/java. By default the Failsafe Maven Plugin looks for integration tests matching the patterns */IT.java, **/IT.java and */*ITCase.java.  You will notice that these bindings do not overlap with the default surefire bindings.  To use the Maven Failsafe Plugin you need to add the following to your pom.xml file.

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.6</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

You will then have the following lifecycle bindings

Phase

Plugin execution goal

test

surefire:test

integration-test

failsafe:integration-test

verify

failsafe:verify

The advantage to using the Maven Failsafe Plugin is that it will not stop the build during the integration-test phase if there are test failures.  The recommendation is that you do not directly invoke the pre-integration-test, integration-test or post-integration-test phases but that instead you run integration tests by specifying the verify phase, e.g.

mvn verify

This allows you to set-up your integration test environment during the pre-integration-test phase, run your integration tests during the integration-test phase, cleanly tear-down your integration test environment during the post-integration-test phase before finally checking the integration test results and failing the build if necessary. Here is an example using jetty for hosting an integration test environment:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.6</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.1.16</version>
        <configuration>
          <scanIntervalSeconds>10</scanIntervalSeconds>
          <stopPort>8005</stopPort>
          <stopKey>STOP</stopKey>
          <contextPath>/</contextPath>
        </configuration>
        <executions>
          <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <scanIntervalSeconds>0</scanIntervalSeconds>
              <daemon>true</daemon>
            </configuration>
          </execution>
          <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

The Future

Rumor has it that a future version of Maven will support something like src/it/java in the integration-test phase, in addition to src/test/java in the test phase.

Surefire

Failsafe

  • No labels

7 Comments

  1. How to you do reporting (such as coverage with cobertura) on core modules using a separate integration test module? Afaik there is no way on instrumenting the core module's classes when they're referenced as an already-packaged jar file.

  2. An alternative to this which is slightly cleaner is to use profiles. Profiles and inheritance are a little buggy so some workarounds were required; however, it can be done as follows:

    1. Create a property to point to your standard test area. 

     <properties>
             <srcdir>src/test/java</srcdir>
     </properties>

    2. Create a profile with a named id such as integration-test. The profile acts to overwrite the source directory - and can also pull in other resources that are integration specific

     <profiles>
      <profile>
       <id>integration-test</id>
           <properties>
             <srcdir>src/integration/java</srcdir>
           </properties>
      </profile>
     </profiles>

    3. In your build stanza, refer to the source directory using the variable

     <build>
       <testSourceDirectory>$

    Unknown macro: {srcdir}

    </testSourceDirectory>
     </build>

    4.To execute in "integration mode", add a -Pintegration-test to your maven call.

    If you place this in your top-level pom it will automatically filter through to the children. One aspect of this configuration is that your non-integration tests aren't executed.

  3. When the unit tests and the integration tests are in the same source directory but not in the same package, there are 3 solutions.

    Solution 1: The solution you explain in shale-usecases

    Solution 2: A solution suggested by John Fergusson in  John Ferguson Smart's Blog

    Solution 3: A solution suggested by Rod Coffin in Wiki: Maven Integration Testing

    Personally  I prefer the solution 3 because it doesn't use a profile (used in solution 1) and it doesn't skip any configuration (strategy used in solution 2).

    What's your opinion?

    Appendix

    A. The structure of the project

     The unit tests are under ch.example.utest.simple, the integration tests are under itest.ch.example.utest.simple

    B. The corresponding POM concerning the solution 1


    C. The corresponding POM concerning the solution 2


    D. The corresponding POM concerning the solution 3



  4. Another alternative is to configure the maven-surefire-plugin exclude using a system property.

    The POM:

    then test as:  

    build as:

    and run integration test as:


    This example assumes that your integration test class names end with IntegrationTest but you could use a directory name convention instead.


     

  5. I followed the solution 2 given by Jean Claude.

    It worked well for almost all my tests, but I encountered a odd problem with test classes having inner classes in it.
    I got an exception like :

    Actually, the configuration :

    Overrides the default behavior of Surefire, that is : ignoring inner classes (Foo$1.class)
    To fix this, you should had another exclude line:

  6. Maven has really good support for JUnit categories now.  You can use them to easily split tests without a naming convention.

    The basic steps are below.  A full working example can be found at the following link.

    http://johndobie.blogspot.co.uk/2012/04/unit-and-integration-tests-with-maven.html

    Its especially good for legacy suites where the tests have not been split at all, but you can also use seperate directories if you wish.

    Define a Marker Interface
    Mark Your Test Classes
    Configure Maven Unit Tests

    mvn clean test will run all your tests which are not marked as an integration tests.

     

    Configure Maven Integration Tests

    mvn clean install will run all of your unit tests and seperately the integration tests in the integration-test phase.

  7. when exceptions occur in the pre-integration-test phase no cleanup is being done (post-integration-test is not executed when pre-integration-test fails).

    Typically in the pre-integration-test phase servers are started or applications are deployed that can be error-prone and might leave the system in a dirty state.

    I'm using the failsafe plugin to ensure that proper cleanup is done when integration tests fail, however, in case where the preperations of these integration tests fail (ex: port in use, app deployment failure) I would like to ensure that the system is properly cleaned up.

    How would you suggest doing the cleanup in that particular case ? Could this be an enhancement of the failsafe plugin or is there another way to handle this scenario ?