Pitest

Name	Pitest Plugin
License	LGPL v3
Authors	Alexandre Victoor
Jira	http://jira.codehaus.org/browse/SONARPLUGINS/component/15475
Sources	http://svn.codehaus.org/sonar-plugins/trunk/pitest
Latest version	0.2
Download	http://repository.codehaus.org/org/codehaus/sonar-plugins/sonar-pitest-plugin/0.2/sonar-pitest-plugin-0.2.jar

Description

PIT is a mutation testing tool for java. You can check out the official pitest web site for more details on mutation testing and PIT.

Long story short, mutation testing is a very smart way to check the relevance of unit tests. The main idea is to alter the tested code and check that at least one unit test fails. An alteration of the code is called a "mutant". A mutant has "survived" the tests if there is no test failure.

The goal of this plugin is to bring PIT results to SonarQube. Right now the integration of these results is quite simple, "survived mutants" on code covered by tests are seen as SonarQubeissues.

Note
Even if PIT detects "survived mutants" on uncovered lines of code, these mutants are simply ignored by the plugin.

Usage & Installation

Limitations of mutation testing

This section is not specific to PIT but since mutation testing is not yet a mainstream method... Here are a couple of general advises and warnings:

Mutation testing is very CPU time expensive. It is really important to control the scope of mutation testing in order to keep acceptable SonarQubeanalysis times. See below for tips on analysis time.
Mutation testing works on true unit tests. Do not try to use it on integration tests, you might mess up your database, file system, whatever external system used by your integration tests.

Henri Coles, creator of PIT, gave very useful tips on the PIT mailing list to help reduce PIT execution time:

Target only specific portions of your codebase (using the class
filters)
Limit the mutation operators used
Limit the number of mutations per class
Tweak the number of threads

Check out the "Advanced configuration properties" at the bottom of this page to address the above points.

SonarQube server installation

On the SonarQube side, just copy the sonar-pitest-plugin jar to the extensions/plugins directory just like any regular SonarQube plugin.

Since mutation testing is not (yet) officially supported by SonarQube, this plugin acts as a "single rule" rule engine... This rule, named "Survived mutant", is disabled by default and hence needs to be activated when pitest is used.

Project build setup

You have two options in order to produce PIT reports for SonarQube. You can either let the plugin run PIT or you can run PIT prior to the SonarQube analysis.

PIT launched by SonarQube

There is a compatibility issue with maven 3.X

This is the recommended way to go if your build is based on maven 2. Most mandatory PIT configuration entries will be set using information provided by the pom.xml file of your project. You just need to activate PIT execution and specify which classes to mutate and tests to execute. Below an example:

Note: you will find at the bottom of this page an exhaustive description of all the configuration parameters.

The example above tells PIT to mutate classes from packages starting with "com.acme.tools.commons". It also specifies that test classes are located in packages starting with "com.acme.tools.commons".

PIT launched before SonarQube

You can launch PIT using the PIT maven plugin or the command line runner. PIT execution must be done before SonarQube analysis. You also need to specify the "reuseReport" mode of the PIT SonarQube plugin.

Pit needs to be configured in order to generate XML reports. Be aware that PIT default behavior is to generate HTML reports. Below a simple configuration example for maven :

inScopeClasses and targetClasses parameters indicated the classes of the system under test where mutations can be performed. In the example above, all the classes from the com.acme.tools.commons package and sub packages may be altered.

Once configured in the maven pom file, you need to run PITusing the following command "mvn org.pitest:pitest-maven:mutationCoverage".

Note : Of course, all the configuration options are clearly explained in the official pitest documentation.

Last but not least, you need to run a SonarQube analysis with the PIT plugin activated in "reuseReport" mode. The following command would do the job :
"mvn sonar:sonar -Dsonar.pitest.mode=reuseReport"

By default SonarQubewill search the latest PIT report in "target/pit-reports". You can specify another location using property "sonar.pitest.reportsDirectory".

You will find below the list of all the available configuration parameters.

Basic configuration properties

Below the exhaustive list of configuration properties of the SonarQube pitest plugin.

Name	Key	Default value	Description
Pitest activation mode	sonar.pitest.mode	skip	Possible values : 'skip', 'active' (not supported yet) and 'reuseReport'
Path to the pitest reports	sonar.pitest.reportsDirectory	target/pit-reports	Path used when mode "reuseReport" is activate to locate pitest xml reports. Pitest creates a new subfolder "timestamp" at each shot. The SonarQube plugin will explore these subfolders and find the newest xml reports generated.
Classes to include in mutation tests	sonar.pit.target.classes	groupId*	The classes to be mutated. This is expressed as a comma seperated list of globs. For example com.mycompany.* or com.mycompany.package., com.mycompany.packageB.Foo, com.partner.. When maven is used, the default value is 'groupId*'
Tests to run	sonar.pit.target.tests	sonar.pit.target.classes	A comma seperated list of globs can be supplied to this parameter to limit the tests available to be run. If not specified, the value of "sonar.pit.target.classes" is used (not recommended)

Advanced configuration properties

About advanced configuration

The properties below mimic the configuration option of the official PIT maven plugin. You can check out the official documentation of this maven plugin to get more detailed information on these options.

Name	Key	Default value	Description
TestNG Groups to include	sonar.pit.included.testng.groups		List of TestNG groups to include in mutation analysis.
TestNG Groups to exclude	sonar.pit.excluded.testng.groups		List of TestNG groups to exclude from mutation analysis.
Throw error if no mutations found	sonar.pit.fail.when.no.mutations	true	Whether to throw error when no mutations found. Defaults to true.
Arguments to pass to child processes	sonar.pit.jvm.args		List of arguments to use when PIT launches child processes. This is most commonly used to increase the amount of memory available to the process, but may be used to pass any valid JVM argument.
Maximum number of mutations to allow per class	sonar.pit.max.mutations.per.class	-1 (no limit)	The maximum number of mutations to create per class. Use 0 or -ve number to set no limit.
Constant amount of additional time to allow for timeouts	sonar.pit.timeout.constant	3000	Constant amount of additional time to allow a test to run for (after the application of the timeoutFactor) before considering it to be stuck in an infinite loop.
Weighting to allow for timeouts	sonar.pit.timeout.factor	1.25	A factor to apply to the normal runtime of a test when considering if it is stuck in an infinite loop.
Mutation operators to apply	sonar.pit.mutators	INVERT_NEGS, RETURN_VALS, MATH, VOID_METHOD_CALLS, NEGATE_CONDITIONALS, CONDITIONALS_BOUNDARY, INCREMENTS	See official PIT documentation for the list of available mutators: http://pitest.org/quickstart/mutators
Mutate static initializers	sonar.pit.mutate.static.initializers	false	Whether or not to create mutations in static initializers. Defaults to false.
Threads	sonar.pit.threads	1	Number of threads to use
Maximum dependency distance	sonar.pit.max.dependency.distance	-1 (no limit)	Maximum distance to look from test to class. Relevant when mutating static initializers
Calls excluded from mutations	sonar.pit.avoid.calls.to	major logging framework APIs	List of packages and classes which are to be considered outside the scope of mutation
Classes not to mutate or run tests from	sonar.pit.excluded.classes		List of globs to match against class names. Matching classes will be excluded from mutation. Matching test classes will not be run (note if a suite includes an excluded class, then it will “leak” back in).
Methods not to mutate	sonar.pit.excluded.methods		List of globs to match against method names. Methods matching the globs will be exluded from mutation.
PIT classpath	sonar.pit.classpath		Not useful when maven is used. Classpath that contains either JUnit or TestNG as well as your code, tests and any dependencies. When maven is used, this classpath is build using maven classpaths.

You can check out the quickstart section of the official pitest web site for detailed instructions.