Skip to end of metadata
Go to start of metadata

Implemented features

  • There are pre-integration-test and post-integration-test phases for preparing and cleaning up integration tests.

Proposed features to implement (short term)

  • A project should be allowed to have both unit tests and integration tests sitting next to each other.
  • Directory structure:
  • There's no need for introducing a third type of tests. Functional tests should be considered integration tests. If there's a need for more types of tests, it should be implemented by adding features to the surefire plugin to configure it in different executions but the 2 phases to attach tests to will be test (for unit tests) and integration-test.
  • Modify the surefire plugin to add a new surefire:it goal. See MSUREFIRE-50.
  • Add new <integrationTestSourceDirectory> to the POM model. See MNG-2009. Note: Does this need updating the minor version of the POM version?
  • Add new phases for integration tests: generate-integration-test-sources, process-integration-test-sources, generate-integration-test-resources, process-integration-test-resources and integration-test-compile. See MNG-2010

Open short-term questions

  • Should it be src/it, src/itest, src/test-integration or src/integration-test?
  • Should the surefire plugin goal be surefire:it, surefire:itest, surefire:integration, surefire:integration-test?
  • Should we add a integration-test-package phase too?
  • Do we add a new it scope (or itest or integration-test)? (see MNG-2011)

Ideas for the future

  • An alternative directory structure:
  • Should we rename src/test into something more explicit like src/test-unit? BTW the same question will arise for the lifecycle which is currently named test even though it's meant to be only for unit tests. Same for <testSourceDirectory>.

Misc

Other issues/articles related to the testing strategy / testing best practices

Older stuff:

  • No labels

15 Comments

  1. <quote>

    Should we add new POM elements? We currently have <testSourceDirectory> for unit tests. Should we add <integrationTestSourceDirectory> and <functionalTestSourceDirectory>?

     

    </quote>
    How about :

     

    I know this would mean changing the POM structure a bit, but IMHO it would be more flexible, and might be better to do it now before we have a huge maven2 following.

    2 reasons that I can think off top of my head:

    1. The above structure should make it easier to extend Test types.
    2. I have come across use cases (a few times now) where Eclipse project tree can have more than one source folder and where it makes little sense to factor out and create two project modules.

    Is this a reasonable change  - what do you think?

  2. I consider that Cactus test on my ejb are unit test. Some, because done inside a container will consider it integration test even if a test affect only one ejb at a time. I think the line between unit/functional/integration is sometimes to thin to base the directory structure on it. Why not have a test directory with inside it a separation by type of test such as:
    test
     |_junit
     |_cactus
     |_httpUnit
     |_JMeter
     |_...
    and so on. The surefire plugin could then call the plugin responsible for cactus, httpUnit....test. It makes the structure clear and simple and the possibility to add test easy.

  3. I really like Rahul suggestion since the division of test can be so different from an one organisation to another. Some might want to separate their tests by frameworks, some might still divide their tests by type but want a way to decompose their junit in two phases of test, ... I think Maven should give some flexibility here and let the user decides. Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

  4. In my mind, the main information that needs to be captured here is the project knowledge regarding acceptable test suite success rate for a successful build.

    Since unit tests verify the semantics of code, much like the compiler verifies the syntax of code, all unit tests must pass at 100%, just as they must all compile.  Integration tests, by definition, test more than one part of the system, and are not required to pass at 100% for the build to be considered successful.  Most likely a release would not be performed until all the integration tests pass at 100%, in fact they might be coded early in the development cycle and then treated as exit criteria for the release.

    So, I'd vote for something like the first approach:

    project/
      |_ src
        |_ main
        |_ test
          |_ java
          |_ resources
        |_ integ
          |_ java
          |_ resources
    

    Then, I would use regular Java packaging to separate the test code for each different technology, such as JMeter, HttpUnit, and so on, if I felt that distinction was important.  Perhaps I might prefer to split these by purpose instead, such as performance, web, and so on.  The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

  5. Two points I'd like to highlight...

    From John R Fallows' comment above I agree to the statement that:

    The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

    From Alexandre Poitras' comment above it makes sense that:

    Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

    Testing Plugins (JUnit/Surefire/JMeter/HttpUnit) could be bound a test -> type (my comment above) with allowance for overrides if required. I believe this would allow Maven to treat any type of 'test' as a 'test' (no special handling) and delegate handling based on its type to the bound plugin(s).

  6. Yeah but my concern is that a tool can be used for more then one type of test. For instance, junit is often used of course for unit testing but also for intregation testing (this is a common case if you use Spring framework). That's why I was proposing something similar to the plugin model but we could use your method for default values, easily overridable in the pom. This way the testing would be consistent with the pom global approach. The test type element would be like the project packaging element, it inferes default value about your tests but those values can be easily overrided by declaring some plugins and binding them to an execution phase. I think it would be a great solution not changing drasticly the pom format.

  7. Here's a little example of what I mean :
    *By the way, forget my comments about having  a tool used for two types of test. I taught you were infering the test type from the tool, but I see now you actually meant test type-> tool. Anyway, what I just proposed still stands.
    <build>
    <sources>
    <source>
    <directory/>
    <output/>
    </source>
    </sources>
    <tests>
    <test>
    <type>Unit/IT/Functional/System/Custom</type>
    <directory/>
    <output/>
    </test>
    </tests>
    <plugins>

    1. let's say you want to override the default tool used for integration test
      <plugins>
      <plugin>
      <groupId>org.junit</groupdId>
      <artifactId>junit</artifactId>
      <version/>
      <extensions/>
      <executions>
      <execution>
      <id/>
      <phase>integration-test</phase>
      <goals/>
      <inherited/>
      <configuration/>
      </execution>
      </executions>
      </plugin>
      </plugins>
      </build>

    What do you thin?

  8. I just thought I'd add to these comments the approach I'm currently using to perform integration tests with M2 as M2 is now. It may ensure that all angles are covered in any of your recommended approaches.

    I have achieved integration testing (using both JunitEE/Cactus and JUnits calling remote interfaces) in M2 using a very simple lightweight POM for each discrete task/execution in the integration phase (i.e. bound to this phase) e.g.

    + application-root  
         pom.xml   (parent pom)
        
       now normal stuff follows e.g. :- 
      
        + normal EAR
        + normal JAR(s)
        + normal WARs
        + test WAR (for JUnitEE or Cactus)

       now follow's the M2 project modules to run the integration tests e.g. :-
      
        + runtests-integration
             pom.xml   (sub-parent pom, invoking child modules in the order below)
            + rebuild-database
                   pom.xml  (antrun to rebuild DB schema)
            + start-container
                   pom.xml  (antrun or cargo to start container )
            + runtests-serverside  (runs junits needing server started as they invoke remote interfaces)
                   pom.xml (surefire:test only picking up tests ending in *ServerTest.java )
            + runtests-incontainer-tests
                   pom.xml (antrun for running JUnitEE or Cactus incontainer tests)
            + stop-container
                   pom.xml (antrun or cargo)
     
    All plugins defined under 'runtests-integration' (and below) are bound to the integration-test phase
     
    I found this approach also necessary, because you can only have one instance of the antRun plugin for the integration-test phase, therefore you can't use antrun, then surefire, then antrun again  as the last antrun overrides the first if they are bound to the same phase.

    I did want to use Cargo, but unfortunately there was no support for OC4J with an existing configuration.

    I have the above 'runtests-integration' suite only execute if the M2 build profile is set to -Pintegrationtests. In this way developers can run a regular M2 build which just runs standalone JUnits, or specify the profile to run a fuller build. We have CruiseControl always running the integrations tests profile.  

    With m2.02 I should now convert the Antruns to Ant Plugins.

    I'm hoping that when M2 supports integration tests better, I can migrate to the approved strategy, I know this approach is a slight misuse of M2, but this works right now, we have 2 large Applications (both EARs) that are built with M2 via CruiseControl, the above integration projects start the container run all the integration tests (with the 2nd EAR application even invoking the 1st EAR's services)

    I have another M2 module in addition to those shown above that actually hold the integration tests :-

        + tests-serverside 
             pom.xml   (JAR pom, building those JUnits needing server started)
            
               The JUnits in this project follow two naming patterns :-
               src/test/java
                     *ServerTest.java  (JUnits needing server started as they invoke remote interfaces )
                     *JEETest.java       (JUnits needing to run inside container via JUnitEE or cactus )

         The above JAR could probably have been split into two separate JARs for each type.
        
         The parent pom.xml overrides the default maven-surefire-plugin by excluding the following :-
        
         <excludes>
          <exclude>**/*ServerTest.java</exclude>
          <exclude>**/*JEETest.java</exclude>
         </excludes>
        
         such that these tests don't get run during the normal test phase. Just regular standalone *Test.java get run.
        

    Hope this is useful

    Pete

     

  9. Bae

    Don't forget to include support for integration testing of plugins.
    This gets more complicated as the plugin has yet to be installed locally (and shouldn't be until it passes the tests)

  10. Hey Pete - thanks for the Comment - you saved my day.

    I've been looking for a way to get my complex integration tests up and running and the only way this was possible was to use the strategy employed by Pete: Have a subproject for each step in your actual integration test lifecylce.

    Right now it's impossible to get your tests up and running in a single module due to the well known bugs. But even if all bugs were fixed it doesn't make a lot of sense with the maven 2 model for the simple reason that there are not enough lifecycle phases and they have the wrong names.

    Complex integration tests require dozens of steps to set up and tear down the testing environment.

    I would propose to make the lifecycle configurable or the ability to just switch it off and run a sequence of plugins like me good old ant file.

  11. As many people point it out, we want a layout similar to

    But if  the integration test directory is ismply called "integ" it will appear before the main directory in most IDE. The chosen name must be such as its comes after "test" in alphabetical order.
    Thanks!

  12. Any news on the separate directory integration-test strategy?

    • I have noticed that lot of test gurus when talking about testing use "unit", "functional", and "scenario". Therefore, another possible alternative directory structure might be:

    Alternatively, the integration might be removed with the assumption that a scenario test will cover integration tests.