Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

There are currently a number of solutions for testing plugins - both in terms of unit testing and integration testing. This has resulted in a confusing situation where plugins can be tested in multiple different ways, and there are no clear instructions for a new plugin author in how to test their plugins.

The objective of this document is to assess all the current approaches, and their positives and negatives, and to provide a baseline for discussion about the various approaches. It will attempt to identify overlap and make proposals to reduce duplication and confusion. It will also attempt to document how to write integration tests for your plugin, depending on your objectives. It will not attempt to propose a single-replacement solution - as there are likely good reasons to have different types of testing, and a separation of different techniques into different plugins (though deprecation of some libraries or techniques to reduce duplication may be necessary).

Available Solutions

Unit Testing

Definition: This is traditional unit testing - it takes no consideration for the fact that it is a Maven plugin, and tests the code itself.

  1. maven-plugin-testing-harness: This library provides assistance for plugin testing by pre-constructing the Plexus components, providing stub objects for Maven functionality such as projects, and populating fields from an XML file that resembles the plugin configuration in the POM.
  2. standard JUnit testing: MOJOs written in Java can be tested using JUnit as standard Java objects (possibly using a mock framework to mock the Maven functionality such as projects). For non-Java MOJOs, a corresponding test framework may be used for that language but the same principles apply. Using the PlexusTestCase can also give some of the instantiation assistance of the testing harness without the stubs.

Integration Testing

Definition: Integration testing for plugins is checking that a plugin works with the Maven framework libraries in a given situation, and is done before the plugin is verified as installable by Maven. It does not necessarily need to do and end-to-end functional test of a Maven project. In the matrix, "contextual integration testing" refers to integration testing the plugin with the Maven framework without running an entire project, where "complete integration testing" refers to testing entire Maven projects before installation of the plugin.

Functional/System Testing

Definition: Functional testing for plugins is running an entire Maven build that utilises a given plugin and verifies the output of the build process.

Smoke Testing

Definition: A form of functional test that would be run in the integration testing phases and checks base functionality for sanity.

Performance Testing

Definition: Performance testing is a specialised form of acceptance testing.

Requirements

Key

(tick)

explicit support in the current implementation

(error)

no support in any way

(green star)

support via Surefire

(red star)

potential support via Surefire

Comparison Matrix

General

 

CIT

INV

ITP

VER

STY

PTH

SMJ

VRP

Has a non-snapshot release

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

Has a non-alpha/beta release

(error)

(tick)

(error)

(tick)

(error)

(error)

(tick)

(tick)

Has an archetype

(error)

(error)

(error)

(tick)

(error)

(error)

(error)

(error)

Can be used for non-Maven testing

(error)

(error)

(error)

(error)

(error)

(error)

(error)

(tick)

Can be used for Maven App testing

(error)

(tick)

(tick)

(tick)

(tick)

(error)

(error)

(tick)

Can be used for plugin testing

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

(tick)

Can be used for report testing

(tick)

(tick)

(error)

(tick)

(tick)

(tick)

(tick)

(error)

Ability to run an htmlunit test after conclusion of execution

Unit Testing Specifics

 

PTH

SFR

Provides stub factories for creating Maven objects

(tick)

(error)

Provides alternate remote repository handling

(error)

(error)

Integration/System Testing - Setup and Verification

 

CIT

INV

ITP

VER

STY

VRP

Able to stage local repository state

(tick)

(error)

(error)

(error)

(error)

(error)

Able to compile custom sources

(error)

(error)

(error)

(error)

(error)

(error)

Able to add custom dependencies

(error)

(error)

(error)

(error)

(error)

(error)

Supports pre-build hook

(green star)

(tick)

(error)

(tick)

(tick)

(error)

Supports verification

(green star)

(tick)

(error)

(tick)

(tick)

(tick)

Supports post-build hook for tearDown

(green star)

(tick)

(error)

(green star)

(tick)

(tick)

Beanshell support

(error)

(tick)

(error)

(error)

(error)

(error)

Groovy support

(error)

(error)

(error)

(error)

(tick)

(error)

Capture Maven exit code and exceptions

(error)

(tick)

(tick)

(tick)

(tick)

(error)

Supports ANSI output

(red star)

(error)

(error)

(red star)

(tick)

(error)

Can ignore failures

(green star)

(error)

(error)

(green star)

(tick)

(error)

Can skip execution

(green star)

(error)

(error)

(green star)

(tick)

(error)

Integration/System Testing - Execution

Note: CIT measures potential ability to use the CIT Java Wrappers to execute - however this appears to not yet be implemented completely. Uncertain features are therefore marked with (question).

 

CIT

INV

ITP

VER

STY

Can stage or use a pre-staged local repository

(tick)

(tick)

(error)

(error)

(error)

Runs with Surefire

(tick)

(error)

(error)

(tick)

(tick)

Generates JUnit XML

(green star)

(error)

(error)

(green star)

(error)

Can measure coverage using standard techniques

(question)

(error)

(error)

(tick)

(error)

Can be used for contextual integration testing

(error)

(error)

(error)

(error)

(error)

Run tests against multiple versions of Maven

(error)

(tick)

(error)

(tick)

(error)

Run tests against multiple versions of the plugin

(error)

(error)

(error)

(error)

(error)

Forked Maven

(question)

(tick)

(error)

(tick)

(tick)

Embedded Maven

(question)

(error)

(tick)

(error)

(error)

Does not pollute source tree on execution

(question)

(error)

(error)

(tick)

(error)

Cleans up anything stored outside target directory

(error)

(error)

(error)

(error)

(tick)

Includes local repository and system temporary directory

Uses unique storage location for all execution files

(tick)

(error)

(error)

(error)

(error)

This is a concurrency issue if it uses the same location.

Isolated from user's environment/settings by default

(question)

(error)

(error)

(error)

(error)

Can be achieved by passing a separate settings file for those that support it.

Able to run from the IDE regular test functionality

(tick)

(error)

(error)

(tick)

(error)

Able to run offline (unless plugin requires it)

(question)

(tick)

(tick)

(error)

(tick)

Offline by default

(error)

(error)

(error)

(error)

(error)

Able to run a single test

(green star)

(tick)

(error)

(green star)

(tick)

Able to run wildcard/regex set of tests

(green star)

(tick)

(error)

(green star)

(error)

Able to run a suite of tests

(green star)

(tick)

(tick)

(green star)

(tick)

Able to run in a separate Maven project

(error)

(tick)

(tick)

(tick)

(tick)

Able to run in the same Maven project

(tick)

(error)

(error)

(error)

(tick)

Fast execution

(question)

(error)

(tick)

(error)

(error)

Maven home is configurable

(question)

(tick)

(error)

(tick)

(error)

Configurable Maven settings file

(question)

(tick)

(error)

(tick)

(tick)

Configurable individual Maven settings

(question)

(tick)

(error)

(error)

(error)

Configurable working directory

(question)

(tick)

(error)

(tick)

(error)

Configurable input/output handlers for Maven

(question)

(tick)

(error)

(error)

(error)

Configurable Maven logger

(question)

(tick)

(error)

(error)

(error)

Can select which projects to run (include/exclude)

(question)

(tick)

(tick)

(tick)

(tick)

Able to run Maven in batch mode

(question)

(tick)

(error)

(tick)

(error)

Able to run Maven in debug mode

(question)

(tick)

(error)

(tick)

(tick)

Able to inherit shell environment

(question)

(tick)

(error)

(tick)

(error)

Pass goals to run Maven with

(question)

(tick)

(tick)

(tick)

(tick)

Pass goals to run Maven with in a file

(question)

(tick)

(tick)

(tick)

(tick)

Interpolate goals to run Maven with

(question)

(tick)

(error)

(error)

(error)

Pass properties to run Maven with

(question)

(tick)

(error)

(tick)

(tick)

Pass properties to run Maven with in a file

(question)

(tick)

(error)

(tick)

(tick)

Interpolate properties to run Maven with

(question)

(tick)

(error)

(tick)

(error)

Pass profiles to run Maven with

(question)

(tick)

(error)

(tick)

(error)

Pass profiles to run Maven with in a file

(question)

(tick)

(error)

(tick)

(error)

Pass flags to run Maven with

(question)

(error)

(error)

(tick)

(tick)

Pass flags to run Maven with in a file

(question)

(error)

(error)

(tick)

(tick)

Able to run tests in parallel

(green star)

(error)

(error)

(green star)

(tick)

Able to timeout tests run in parallel

(green star)

(error)

(error)

(green star)

(tick)

Suggested Use

Info
titleTODO

To be completed once matrix is complete and examples have been generated to decide direction

Proposals

Info
titleTODO

To be completed based on suggested use and missing needs

  • dfabulich: Make maven-verifier share code with maven-invoker. maven-verifier duplicates a number of utilities; its Maven launcher is entirely surpassed by what maven-invoker can do. Verifier should be just a simple library for configuring and verifying maven-invoker runs, intended to be used in JUnit/TestNG.

Notes

  • IT plugin has grown out of date
  • verifier is stable - used in Maven ITs and Surefire ITs
  • invoker is stable
  • plugin testing harness is stable but alpha
  • plugin testing tools/component IT plugin is stable but alpha

John says: "I'm hoping to converge the tools
we're using in the core ITs with those in the maven-component-test-plugin
(used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and
arrive at some single solution (a plugin) that we can use everywhere...but I
haven't even been able to come up with a unified feature list yet."

Plugin Testing Tool documentation discusses the need for a different local repository (need could be reviewed by improving local repo mechanism, so keep that functionality separate). Incorporate doc into final solution. Note disagreement with "test suite maintenance" - integration tests != functional tests, and both should be well designed, not easily adapted from users (though easily executing users's tests and helping them construct well designed tests is a better goal). Note: Not much different between component tool and plugin tool

Test without installation (c.f. staging and unstaging of plugin - implications for concurrent execution) - to Not pollute the local repository (how about using a timestamped version?

Invoker Plugin has a lot of logic in one mojo, and duplicates other plugins

IT plugin is now duplicated with invoker

Plugin Management Plugin seems roughly equivalent to the component IT plugin but for general use. Could be irrelevant under local repository changes.

Test Tools re a library used by component IT - does a couple of file manipulations for tests.

Verifier duplicates a lot of utilities

Usages

Assembly Plugin

Runs within src/it, cleans within there
Functional tests are not included in the POM.

Dependency Plugin

Integration tests are not integrated - set up as separate test projects.

Surefire Plugin

Using the separate project with standard Maven IT practices (these are really functional tests?)

Project Info Reports

Does HTMLUnit tests on the output - manually constructs the mojo like a regular IT, uses plugin testing harness.
Quite slow, though.
Adjustments should be revised.

Other Executors

Some other projects implement Maven executors (eg the release plugin). These should simply be migrated to the invoker (adding in any additional functionality as needed) rather than seeking a unified solution at this point. It is unrelated to testing.

Resources

Discussions:

Plugins and libraries for testing: