Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Draft

Icon

I am in the process of researching and documenting all the current solutions offline and tying together a more complete proposal that matches the criteria below. - Brett 26/Dec/07

Overview

There are currently a number of solutions for testing plugins - both in terms of unit testing and integration testing. This has resulted in a confusing situation where plugins can be tested in multiple different ways, and there are no clear instructions for a new plugin author in how to test their plugins.

The objective of this document is to assess all the current approaches, and their positives and negatives, and to provide a baseline for discussion about the various approaches. It will attempt to identify overlap and make proposals to reduce duplication and confusion. It will also attempt to document how to write integration tests for your plugin, depending on your objectives. It will not attempt to propose a single-replacement solution - as there are likely good reasons to have different types of testing, and a separation of different techniques into different plugins (though deprecation of some libraries or techniques to reduce duplication may be necessary).

Available Solutions

Unit Testing

Definition: This is traditional unit testing - it takes no consideration for the fact that it is a Maven plugin, and tests the code itself.

  1. maven-plugin-testing-harness: This library provides assistance for plugin testing by pre-constructing the Plexus components, providing stub objects for Maven functionality such as projects, and populating fields from an XML file that resembles the plugin configuration in the POM.
  2. standard JUnit testing: MOJOs written in Java can be tested using JUnit as standard Java objects (possibly using a mock framework to mock the Maven functionality such as projects). For non-Java MOJOs, a corresponding test framework may be used for that language but the same principles apply. Using the PlexusTestCase can also give some of the instantiation assistance of the testing harness without the stubs.

Integration Testing

Definition: Integration testing for plugins is checking that a plugin works with the Maven framework libraries in a given situation, but does not do and end-to-end functional test of a Maven project.

  1. ...

Functional Testing

Definition: Functional testing for plugins is running an entire Maven build that utilises a given plugin and verifies the output of the build process.

  1. ...

Smoke Testing

Definition: A form of functional test that would be run in the integration testing phases and checks base functionality for sanity.

Performance Testing

Definition: Performance testing is a specialised form of acceptance testing.

State of the Nation

  • IT plugin has grown out of date
  • verifier is stable - used in Maven ITs
  • invoker is stable
  • plugin testing harness is stable but alpha
  • plugin testing tools/component IT plugin is stable but alpha

John says: "I'm hoping to converge the tools
we're using in the core ITs with those in the maven-component-test-plugin
(used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and
arrive at some single solution (a plugin) that we can use everywhere...but I
haven't even been able to come up with a unified feature list yet."

Plugin Testing Tool documentation discusses the need for a different local repository (need could be reviewed by improving local repo mechanism, so keep that functionality separate). Incorporate doc into final solution.

Test without installation (c.f. staging and unstaging of plugin - implications for concurrent execution)

Not pollute the local repository (how about using a timestamped version?)

Invoker Plugin has a lot of logic in one mojo, and duplicates other plugins

Requirements

Key

CIT

maven-component-it-plugin

CIJ

maven-component-it-plugin with Java wrappers generated

INV

maven-invoker-plugin

ITP

maven-it-plugin

VER

maven-verifier

STY

shitty-maven-plugin

PTH

maven-plugin-testing-harness

Comparison Matrix

 

CIT

CIJ

INV

ITP

VER

STY

PTH

Runs with Surefire

(error)

(tick)

(question)

(question)

(question)

(question)

(question)

Generates JUnit XML

(error)

(tick)

(question)

(question)

(question)

(question)

(question)

Can measure coverage using standard techniques

(error)

(tick)

(question)

(question)

(question)

(question)

(question)

Can be used for unit testing

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Can be used for integration testing

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Can be used for functional testing

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Has a non-snapshot release

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Has a non-alpha release

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Has an archetype

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Test without installation

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Run tests against multiple versions

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Forked Maven

N/A

N/A

(tick)

(error)

(question)

(question)

(question)

Embedded Maven

N/A

N/A

(error)

(tick)

(question)

(question)

(question)

Beanshell support

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Groovy support

N/A

N/A

(error)

(error)

(error)

(tick)

(error)

Suite setup steps

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Suite teardown steps

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Suite verification steps

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Does not pollute source tree on execution

(error)

(error)

(error)

(question)

(question)

(question)

(question)

Does not pollute the local repository

(tick)

(tick)

(error)

(question)

(question)

(question)

(question)

Does not pollute the temporary directory

(error)

(error)

(tick)

(question)

(question)

(question)

(question)

Isolated from user's environment/settings

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Able to run from the IDE

(error)

(tick)

(question)

(question)

(question)

(question)

(question)

Able to run offline (unless plugin requires it)

(tick)

(tick)

(tick)

(question)

(question)

(question)

(question)

Offline by default

(error)

(error)

(error)

(error)

(error)

(error)

(error)

Can be used for testing non-Maven things

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Can be used for Maven testing

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Can be used for plugin testing

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Can be used for report plugin testing

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Tests plugin annotations

N/A

N/A

(question)

(question)

(question)

(question)

(question)

Able to run a single test

N/A

N/A

(question)

(question)

(question)

(question)

(question)

Able to run a regex set of tests

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Able to run a suite of tests

N/A

N/A

(question)

(question)

(question)

(question)

(question)

Able to drop-in an IT easily from a user submission

(tick)

(error)

(question)

(question)

(question)

(question)

(question)

Able to compile custom sources

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Able to add custom dependencies

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Able to run in a separate project

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Able to run in the same project

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Provides stub factories for creating Maven objects

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Provides alternate remote repository handling

(error)

(error)

(question)

(question)

(question)

(question)

(question)

Fast execution

N/A

N/A

(question)

(question)

(question)

(question)

(question)

Able to stage local repository state

(tick)

(tick)

(question)

(question)

(question)

(question)

(question)

Capute Maven exit code and exceptions

N/A

N/A

(tick)

(tick)

(question)

(question)

N/A

Maven home is configurable

N/A

N/A

(tick)

(error)

(question)

(question)

N/A

Configurable Maven settings file

N/A

N/A

(tick)

(question)

(question)

(question)

N/A

Configurable individual Maven settings

N/A

N/A

(tick)

(error)

(question)

(question)

N/A

Configurable working directory

N/A

N/A

(tick)

(question)

(question)

(question)

(error)

Configurable input/output handlers for Maven

N/A

N/A

(tick)

(question)

(question)

(question)

(error)

Configurable Maven logger

N/A

N/A

(tick)

(question)

(question)

(question)

(error)

Can select which projects to run (include/exclude)

(tick)

(tick)

(tick)

(question)

(question)

(question)

(error)

Able to run Maven in batch mode

N/A

N/A

(tick)

(question)

(question)

(question)

(error)

Able to run Maven in debug mode

N/A

N/A

(tick)

(question)

(question)

(question)

(error)

Able to inherit shell environment

N/A

N/A

(tick)

N/A

(question)

(question)

N/A

Pass goals to run Maven with

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Pass goals to run Maven with in a file

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Interpolate goals to run Maven with

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Pass properties to run Maven with

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Pass properties to run Maven with in a file

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Interpolate properties to run Maven with

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Pass profiles to run Maven with

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Pass profiles to run Maven with in a file

N/A

N/A

(tick)

(question)

(question)

(question)

(question)

Techs

IT Plugin

Runs a series of POMs
Uses the embedder
Not as feature rich as the invoker
Some duplication

Plugin Management Plugin

Roughly equivalent to the component IT plugin but for general use. Could be irrelevant under local repository changes.

Test Tools

Used by component IT - does a couple of file manipulations for tests.

Verifier

Duplicates a lot of utils
Encompasses a lot of needed requirements for forked execution
Used in core/surefire ITs

Verifier Plugin

Separate from verifier
Runs similar verification steps, bound to the lifecycle
Loaded from XML file
Supports regex of file, existence of file
Not overly useful - probably better to use scripts

*ITTY Plugin

Removes all integration test module output and logs.
Installs the current modules artifact with the version testing , so that the test can be 100% which artifact is being used.
Invoke child Maven processes on each of the integration test modules and capture the results.
Groovy pre/post hooks
Runs in integration test phase
Could use surefire for some other parts
Runs a set of pre-defined goals
Installs to standard local repository with version "testing"
Pollutes source directories
Select test to run (duplicates surefire configuration)
Can run tests in parallel (duplicates TestNG surefire configuration), supports timeout
Can pass offline, debug mode to tests
Supports ANSI output (should be integrated into Surefire)
Can pass arbitrary properties (via file)
Can pass arbitrary Maven flags (as arg, or via file)
Can ignore failures and skip execution
Downside: name
Forks Maven

Plugin Testing Tools

Trigger Maven builds using test projects, without incurring any sort of conflict with plugins already in use within the current (this plugin's) build.
Construct a local repository into which the test version of the plugin can be installed, allowing the main local repository to remain untouched by the tests. Integration tests can be directed to use this test-time local repository instead of the normal location.
Support tracking and analysis of results from Maven builds spawned during integration testing. As far as is reasonable, test builds should be configured in the same way as normal project builds, to make the most of test cases submitted by users along with the issues they file.
Contains an Invoker delegate

  • Note disagreement with "test suite maintenance" - integration tests != functional tests, and both should be well designed, not easily adapted from users (though easily executing users's tests and helping them construct well designed tests is a better goal)
  • Note: needs to be more specific about the goals
  • Note: Not much different between component tool and plugin tool

Usages

Assembly Plugin

Runs within src/it, cleans within there
Functional tests are not included in the POM.

Dependency Plugin

Integration tests are not integrated - set up as separate test projects.

Surefire Plugin

Using the separate project with standard Maven IT practices (these are really functional tests?)

Project Info Reports

Does HTMLUnit tests on the output - manually constructs the mojo like a regular IT, uses plugin testing harness.
Quite slow, though.
Adjustments should be revised.

Other Executors

Some other projects implement Maven executors (eg the release plugin). These should simply be migrated to the invoker (adding in any additional functionality as needed) rather than seeking a unified solution at this point. It is unrelated to testing.

Proposals

Resources

Discussions:

Plugins and libraries for testing:

  • No labels