best practices - testing strategies

Implemented features

• There are pre-integration-test and post-integration-test phases for preparing and cleaning up integration tests.

Proposed features to implement (short term)

• A project should be allowed to have both unit tests and integration tests sitting next to each other.
• Directory structure:

project/
  |_ src
    |_ main
    |_ test
      |_ java
      |_ resources
    |_ it
      |_ java
      |_ resources

• There's no need for introducing a third type of tests. Functional tests should be considered integration tests. If there's a need for more types of tests, it should be implemented by adding features to the surefire plugin to configure it in different executions but the 2 phases to attach tests to will be test (for unit tests) and integration-test.

• Modify the surefire plugin to add a new surefire:it goal. See MSUREFIRE-50.

• Add new <integrationTestSourceDirectory> to the POM model. See MNG-2009. Note: Does this need updating the minor version of the POM version?

• Add new phases for integration tests: generate-integration-test-sources, process-integration-test-sources, generate-integration-test-resources, process-integration-test-resources and integration-test-compile. See MNG-2010

Open short-term questions

• Should it be src/it, src/itest, src/test-integration or src/integration-test?
• Should the surefire plugin goal be surefire:it, surefire:itest, surefire:integration, surefire:integration-test?
• Should we add a integration-test-package phase too?
• Do we add a new it scope (or itest or integration-test)? (see MNG-2011)

Ideas for the future

• An alternative directory structure:

project/
  |_ src
    |_ main
    |_ test
      |_ unit
        |_ java
        |_ resources
      |_ integration
        |_ java
        |_ resources
      |_ functional
        |_ java
        |_ resources

• Should we rename src/test into something more explicit like src/test-unit? BTW the same question will arise for the lifecycle which is currently named test even though it's meant to be only for unit tests. Same for <testSourceDirectory>.

Misc

Other issues/articles related to the testing strategy / testing best practices

• http://jira.codehaus.org/browse/MNG-1922: Rename maven-it-plugin plugin as its name implies it is for performing integration tests in general
• http://jira.codehaus.org/browse/MNG-1381: best practices: testing strategies
• http://jira.codehaus.org/browse/MNG-591
• http://docs.codehaus.org/display/MAVENUSER/Review+of+Plugin+Testing+Strategies

Older stuff:

• Testing Strategies