Maven Plugin and Extension Loading Design

Since its 2.0 release, Maven has had two modes of extension. The first, its plugin framework, is what provides the flexibility to execute collections of plugins that actually perform the build tasks for a project. These tasks are handled by plugins in order to allow recombination of the tasks in order to satisfy nearly any type of project build, and they depend on Maven to keep their respective classpaths independent from one another, to prevent incompatibilities.

The second extension mode in Maven 2 is what we know as build extensions. These are artifacts injected into a project's build that provide custom implementations of components that help Maven's core manage the runtime representation of projects. They might include Maven Wagon implementations, to provide access to artifacts and metadata over special network transports; or, they might contain custom plugin mappings (also known as lifecycle mappings) and artifact handler components, geared toward special project packaging types.

Since the Maven 2.0 release, we've learned a lot about what works and what doesn't in the way Maven 2.0.x conceives of these different extension frameworks. In developing Maven 2.1.x, it's time to take a look at what we can improve.

Existing Design - Maven 2.0.x

Plugins

Overview

Plugins in the Maven 2.0.x design are separated from the core Maven classloader by the PluginManager. It creates and manages separate PlexusContainer instances for each plugin, which are child containers of the core Maven PlexusContainer instance. This allows plugin artifacts to be added to the child container, where they will be isolated for that plugin's use only.

Plugin-Tracking Design Flaws

In addition to container management, the PluginManager and its associated components are responsible for tracking plugins in the runtime, so they can be reused in the event that multiple mojos are used from a single plugin, or multiple projects in the same reactor make use of the same plugin. In 2.0.x, plugins are tracked by the key: {groupId, artifactId}, which doesn't take into account the possibility that projects may declare different versions or plugin-level dependencies for a given plugin. This exposes two critical design problems in Maven 2.0.x. First, that a hierarchy of projects using two different versions of a given plugin will behave inconsistently between single-project builds of its modules and a single, multimodule build of the whole hierarchy. Since this is much simpler to talk about concretely, let's consider an example.

When projects A and B are both listed as modules (in that order) of some top-level project T, they can be built from either the T level (where both A and B will normally be built), or individually. Imagine project A declares that it uses the maven-assembly-plugin version 2.0, and binds the 'attached' goal to its lifecycle. Then project B declares that it also depends on the maven-assembly-plugin, but this time it depends on version 2.1, and binds the 'single' goal to its lifecycle. When project A is built individually, maven-assembly-plugin version 2.0 executes as part of the build, and the project is built successfully. When project B is built individually, maven-assembly-plugin version 2.1 is used, and project B's build likewise succeeds. However, when project T is built, things change. Project A is built as the first module build after T itself succeeds (A is listed first, and there is no interdependency between A and B, so A is built ahead of B); project A's build succeeds. Next, project B is built, and fails. This is because project A already prompted Maven to load maven-assembly-plugin, version 2.0, so it could run the 'attached' mojo. When project B comes along expecting Maven to have loaded maven-assembly-plugin, version 2.1, it fails to locate the 'single' mojo (assembly:single wasn't introduced until version 2.1) and the build fails.

The second major design flaw exposed by {groupId, artifactId} tracking of plugins is that it cannot track multiple plugin-level-dependency profiles for a single plugin in the runtime. To illustrate, consider again a three-project hierarchy, T, A, and B. T lists A and B as modules in that order. Project A declares that it uses the maven-antrun-plugin, with no plugin-level dependencies. Project B also declares that it uses the maven-antrun-plugin, but it adds plugin-level dependencies on ant-commons-net, commons-net, and oro, which are necessary to run the 'telnet' Ant task. When built individually, both project A and project B will succeed. However, when build from the level of project T, we see a similar problem to the aforementioned. Project A's build succeeds, and in doing so, prompts Maven to load the maven-antrun-plugin with NO plugin-level dependencies. When project B builds, it expects Maven to consider the plugin-level dependencies it declares for the antrun plugin, and naively uses the 'telnet' Ant task. Since Maven used the plugin-level dependencies from project A when loading the antrun plugin (there were none), it doesn't have access to the 'telnet' Ant task, and project B's build fails.

Both of these failing use cases expose an interesting effect in Maven 2.0.x: that the success or failure of a multimodule build can depend entirely on the interproject dependencies and module ordering that affect the order in which reactor projects are built, and therefore, the information used to initialize the plugins that build them.

Artifact Filtering

Aside from the problems related to plugin versions and plugin-level dependencies, which are actually related to the way in which the PluginManager tracks plugins, the design of Maven 2.0.x also exposes the potential for plugin-core class incompatibilities. This arises from the fact that Maven filters the dependencies resolved for its plugins, to exclude those artifacts that are already present in the core classloader. This filtering step does not take into account the versions of core artifacts specified by plugins or provided by the core, which implies that a plugin could rely on maven-project version 2.1, yet still be executed inside a Maven 2.0.8 runtime. Obvious incompatibilities like NoSuchMethodError can occur from this practice.

Maven 2.0.x mitigates this potential for incompatibility using the 'prerequisites/maven' section of the POM, in which the minimal version of Maven with which the plugin is compatible is stated. However, this minimal value makes no allowance for an upper limit. As APIs are deprecated then removed over the course of successive Maven releases, these old plugin versions could become incompatible without the system recognizing that fact.

NOTE: This has been at least partially alleviated in 2.1.x by allowing version ranges for the 'prerequisites/maven' section of the POM, which allow both lower and upper bounds on the compatibility statement.

Plugin Extensions-Flag Limitations

Finally, Maven 2.0.x provides severe restrictions for plugins that use the extensions flag. The intention of this flag is to allow plugins to also contribute custom component instances and lifecycle mappings. However, the way 2.0.x actually uses the extensions flag limits plugin component contributions to custom artifact handlers and lifecycle mappings - in short, only custom project and dependency types can be contributed by way of a plugin with extensions == true.

To make matters worse, by the time a plugin is found to have extensions enabled, all of the projects in the reactor have already been instantiated. This means the Artifact instances these projects contain (and which will be used to install and deploy the project's artifacts) contain references to ArtifactHandler instances that may have been created on-the-fly in response to an unrecognized packaging type, all because the plugin that provided that ArtifactHandler implementation had not been loaded yet.

The net effect is that plugins using the extensions flag are only able to provide lifecycle mappings for projects in the reactor, and ArtifactHandler instances for use in resolving dependencies of projects in the reactor. Plugins that would bring in legitimate, alternative implementations of core Maven components for use throughout the project build are prohibited from doing so.

Build Extensions

Build extensions in the Maven 2.0.x design are loaded into one of two places, based on some relatively arbitrary (at least, from the perspective of a Maven user) rules. Depending on the conditions specified below, extension classes will either be added to a special 'extension' child PlexusContainer instance, or added directly to the core Maven PlexusContainer instance.

Extensions with Lifecycle Mappings

If a build extension or one of its dependency artifacts contains a lifecycle mapping, that extension's entire dependency chain is added to the core classloader (except for those dependencies filtered out because they're already present in the Maven core). This allows the LifecycleExecutor to lookup these mappings when it tries to build a project with that custom packaging (lifecycle mappings are keyed by a project packaging, and specify the basic steps required to build that sort of project).

Extensions with 2 or Fewer Total Artifacts, Including the Extension Artifact Itself

NOTE: Recently (in 2.0.7 or 2.0.8), the plexus-utils library was allowed to vary independently for each plugin. This caused problems with plugins that depended on plexus-utils to be in the core, and had therefore incorrectly stated their actual dependencies in their POMs. To mitigate this problem, plexus-utils version 1.1 is injected into any extension dependency chain that doesn't already include a plexus-utils version. This is why the rules below make special consideration for extensions with 2 or fewer artifacts in their classpath, instead of limiting it to 1 artifact.

If the extension doesn't contain lifecycle mappings and its total classpath consists of 2 or fewer artifacts, that extension's entire classpath is again added to the core classloader. This is done to account for the original assumption that certain plugins may have configuration files or other custom-configuration classes that are made available to the project build through extensions. It was done to preserve functionality for builds already taking advantage of this in the wild after the 2.0 release.

Extensions with More than 2 Total Artifacts

If an extension's total classpath consists of more than 2 artifacts, that extension is assumed to fill the third common use case for build extensions: Wagon implementations. (The first two use cases are lifecycle mappings and artifact handlers, and are meant to be accommodated by the first rule, above). These extensions are added to a global child PlexusContainer instance called 'extensions', and any Wagon implementations are later mapped to the extensions container inside the WagonManager component for reference during the build. It appears that no other code in Maven's core makes use of the extensions-specific child container, and only the Wagon use case is supported for extensions with a classpath longer than 2 artifacts.

The main reasoning for this isolation is to prevent Wagon implementations from dragging other APIs into the core classloader that will later cause incompatibilities with certain plugins in the build. A classic example of this is the Maven SCM Wagon, which depends on Maven SCM (used to access systems like Subversion and CVS). If Maven SCM were added to the core classloader and specified an old version for this artifact, plugins like the maven-release-plugin may encounter runtime exceptions based on incompatible classes in the older library.

Why These Rules Are Inadequate

Despite the best of intentions, these rules are still hopelessly inadequate for preserving the viability of a build. There are several obvious ways in which careless or complex build extensions could result in critical incompatibilities after Maven allocates them using the logic above.

For example:

Add plexus-utils version 1.0 as a build extension.

There is no logical reason to do this, of course, but you could substitute any other common library that has no dependencies. Users might opt to add these in order to influence which implementation is used in a plugin/framework combination that utilizes the jar service mechanism for lookups. If multiple of these sorts of plugins are in play, this would be a logical thing to attempt, since it centralizes configuration of the component implementation you want to use.

Since plexus-utils is included in the classpath of the above example (or, could be injected into the classpath of any zero-dependency library without exceeding 2 total artifacts), this will result in plexus-utils or our theoretical library being added directly to the core classloader. In the case of plexus-utils, many many plugins depend on this library, and incompatibility is virtually inevitable with such an old version.

Add an extension that specifies a lifecycle mapping and depends on maven-wagon-scm.

A user who is not experienced in the nuances of the above rule-set may create a new lifecyle mapping and associated artifact handler to deal with his special type of project. If the lifecycle mapping refers to a plugin that depends on wagon-scm to do its job, it is logical that wagon-scm be added as a dependency of the extension that declares the lifecycle mapping, so that the two are always added at the same time...thereby reducing the chance of human error where one is added but the other is not.

In this case, since the extension contains a lifecycle mapping, it entire classpath will be added to the core classloader...including maven-scm itself. If the version of maven-scm used by wagon-scm is old, the chances are good that this addition will eliminate the possibility that the maven-release-plugin can be used to release the project.

Appearance of Inconsistency from User's Perspective

Since the rules laid out above will allow certain component definitions to escape into the core classloader while blocking others, the user who adds build extensions with the hope of accessing the custom components defined within will find the behavior of Maven 2.0.x wonky at best. Particularly in the case of the 2-artifact rule, without specific knowledge of the logic encoded in the 2.0.x ExtensionManager, the use of build extensions will seem like a trial-and-error process for each new extension added to the build.

All Extensions Available to All Projects: Potential for Inconsistent Builds

Besides the appearance that the Maven 2.0.x extension mechanism works only sometimes (depending on what you try to do with it, of course), it also introduces yet another inconsistency into Maven's build functionality, depending on whether the build is a multimodule or single-project execution. This arises from the fact that Maven relegates all multi-artifact build extensions into a single child PlexusContainer instance, regardless of how many projects are in the current reactor. This opens up the possibility that one project's extensions can pollute another project's build when the two are processed as part of the same multimodule build, while they would each build successfully if built individually.

For example, suppose a top-level project T lists two modules, project A and project B, in that order. Project A declares a build extension of wagon-foo-1.0, and project B declares a build extension of wagon-foo-1.1. Individually, builds for both project A and project B will complete successfully. However, when project T is built, things change. Project A's build extension - wagon-foo-1.0 - gets loaded into the extension container, as A's build eventually completes successfully. When project B builds, its build extension - wagon-foo-1.1 - is also loaded, but is farther down the classpath list than wagon-foo-1.0, so wagon-foo-1.0 has classloader priority. If wagon-foo-1.0 is incompatible with project B's build, project B's build will fail...but ONLY in this multimodule scenario. Building B by itself again will still yield success.

NOTE: Other examples of this sort of incompatibility can be found by considering two projects that use two different lifecycle mappings and/or artifact handlers for the same packaging type. Admittedly, these are rare concerns, but the global nature of the extension container has great potential to cause all sorts of problems in multimodule builds that don't exist in single-project builds.

Goals and Priorities for Refactoring

Top-Priority Goals

Plugin Loading

Extension Loading

Other Goals

Maven 2.1.x Design

Principles

  1. Extensions (and Plugins with Extensions == true): Each extension should have its own isolated classloader that inherits from the core Maven classloader. Extension classloaders should be keyed (for lookup and reuse) by the extension coordinate ({groupId, artifactId, version}).
  2. Projects: Each project that contains either build extensions or a plugin that uses the extensions flag should have its own classloader that inherits from the Maven core classlaoder. This project-specific classloader should be keyed (for later lookup) by the project coordinate (again, {groupId, artifactId, version}). These project-specific classloaders are primarily orchestration points where a restricted set of extension classes can be imported from the isolated extension realm mentioned above.
  3. Plugins: Each plugin should have its own classloader that does not inherit from the core Maven classloader. Plugin classloaders should be keyed (for lookup and reuse as successive mojos execute) using both the plugin's coordinate ({groupId, artifactId, version}) and a hash of the coordinates of all first-level dependencies for that plugin, including plugin-level dependencies. Parent classloaders for the plugin's classloader should be assigned to just before and nullified just after a plugin executes, to allow flexibility for use with different project-specific classloaders, and to avoid any pollution from previous executions into successive ones.

Foundational Work

Some key elements in Maven's dependency libraries have been overhauled since the versions used in 2.0.x, which provide crucial functionality for the 2.1.x plugin- and extension-loading refactor:

Critical Classworlds (and Classworlds Exposure) Changes

Starting in plexus-container-default version 1.0-alpha-16, the underlying classworlds implementation was overhauled. Additionally, PlexusContainer itself was overhauled, with the new version allowing much better control over which ClassRealms were used to load components. Now, the container tracks which realm is used to load which components, and allows the user to set a global "lookup realm" to be used as the main realm for accessing components during subsequent lookup calls. The container also allowed specification of a lookup realm for each lookup method call. All of this had the effect that subsets of the realms available to a container instance could be isolated for looking up components at different points in the calling code, which amounts to much finer-grained control over classloading in the container instance.

Critical Plexus (plexus-container-default) Changes

Starting with the release of plexus-container-default version 1.0-alpha-35, the PlexusContainer interface was extended one step further to allow even better control over the realms available for component-loading from the container instance. This new method, removeComponentRealm(..), allows the calling code to trigger the separation of a ClassRealm instance from the container. The separation process entails disposing all components from the realm in question, then removing all traces of the realm from the various container- and component-state managers in the system. This new feature allows long-lived Plexus applications to pare down the realms in play to only those that are still active.

As an example usage, the MavenEmbedder now takes advantage of this feature by clearing out all plugin, project, and extension ClassRealm instances after a build completes, thereby freeing significant memory resources for the next execution. This was a crucial addition for those who embed Maven in other applications, such as IDEs, since it eliminated what was in effect a memory leak tied to plugin- and extension-loading that had a history all the way back to Maven 2.0.

Implementation Details

New Components

ExtensionManager Changes

As always, the ExtensionManager is responsible for resolving and managing access to build extensions for the rest of the Maven runtime. However, in 2.1.x the strategies used for extension management have changed significantly. First, the global extension container has been discontinued. Second, the three-tiered rule set used to determine where in the classloader hierarchy different extensions would be loaded has been removed.

Now, build extensions are tracked for exposure to the runtime according to what project they're associated with. As outlined above in the Principles Section, each extension gets its own ClassRealm instance keyed by its coordinate, and only a restricted set of classes from the extension are actually exposed to the runtime through the project-specific ClassRealm instance used for orchestration. This all happens inside the MavenRealmManager, with extension resolution happening in the ExtensionManager itself. Currently, only the implementation classes of those components defined in the extension artifact itself are imported into the project realm, with all other classes remaining in the isolated extension ClassRealm, available for use only by component instances that come from that realm.

Extension ClassRealm instances are never duplicated. Instead, the ExtensionManager checks for the existence of a realm for the extension artifact before resolving it; if the realm already exists, the ExtensionManager reuses it by re-importing the aforementioned class set into the new project realm (keyed to the current project declaring it). Therefore, two projects declaring the same build extension will actually share the same extension ClassRealm (and, depending on instantiation strategies used for components within, possibly even the same component instances).

Plugins that use the extensions flag are treated in much the same way within the ExtensionManager, with some notable differences. First, since the declaration in question comes from the plugins section of the POM, it's possible that there is no version defined. This means that the ExtensionManager must first delegate to the PluginManager to have the plugin loaded, so as to reuse the loading logic used for other plugins and avoid inconsistencies. The PluginManager requires a MavenProject instance and an Artifact instance in order to do this, so the ExtensionManager actually uses the primitive Model parameter that was passed in to create a dummy MavenProject instance, then uses an ArtifactFactory component to create an Artifact for the plugin, before calling the PluginManager.verifyPlugin(..) method. One side effect of this approach is that two ClassRealms are actually created for the plugin with extensions == true: one in the PluginManager, which will be used to execute the plugin, and another in the ExtensionManager, to allow importing extension components into the project realm.

NOTE: While it's possible that such duplication of realm tracking for plugins with extensions could conceivably lead to problems - like ClassCastException - where the class is loaded from both the extension and plugin realms, the chances of this are vanishingly small, particularly when you consider that only component-implementation classes are currently exposed from the extension realm.

Tidying Up

The final notable change for 2.1.x is implemented in the MavenEmbedder. As describe briefly in the Foundational Work section above, the embedder now cleans up after itself after each execution, by prompting the removal of all plugin, extension, and project realms from the core container. This effective erases any modifications made to the Maven runtime for the purposes of the completed build, leaving a clean system for subsequent executions. The actual call is to MavenExecutionRequest.clearAccumulatedBuildState(), which cascades through MavenRealmManager.clear(), on to PlexusContainer.removeComponentRealm(..) (which is called for each realm tracked by the realm manager). Again, since ClassRealm instances tend to consume a relatively large chunk of memory, this is a significant step toward making the embedder usable in long-lived processes such as IDEs.

Existing Issues / Areas for Improvement

While all of the issues outlined below have the potential to cause counter-intuitive build failures, those marked as merely IMPORTANT are considered edge cases that should not affect a wide spectrum of users.

[CRITICAL] Plugin Extensions and POM Inheritance

Plugins that use the extensions flag could conceivably cause issues for the ExtensionManager, since their coordinate information (most notably, their version) can be inherited from parent POMs or injected from a pluginManagement section of the POM. Additionally, plugin-level dependencies may be inherited or injected in a similar fashion. Currently, I'm not entirely confident that these use cases have been covered in tests. If they are broken, some additional logic will be required to ensure some minimal inheritance and managed-data injection take place before the extension plugin is initialized.

[CRITICAL] Plugins with extensions flag and plugin-level dependencies will experience the same first-come-first-served behavior as all plugins did in Maven 2.0.x.

One possible solution to this is to simply adopt the same realm key algorithm used in non-extension plugins currently in 2.1.x, though the GOTCHA about inheritance and injection of plugin information still applies here.

[IMPORTANT] Support builds that specify interproject extension references

I think this issue creates a backward-compatibility problem from 2.0.x, though I can't find a test that verifies it.

The simplest example of this issue goes as follows: top-level project T specifies modules for project A and project B. Project B uses project A as a build extension. None of these artifacts exists in the Maven repository. Building from T should complete successfully. Currently, since all extensions are discovered in the pre-scanning step prior to project instantiation, the build would fail with an ArtifactNotFoundException, since A is not in the repository and Maven doesn't know that it exists in the current reactor, or how to put off the extension-loading step for B until A's build completes.

Since extension-scanning happens even before the first MavenProject instance is constructed, the project-set for the current build has not been established, much less sorted. This means it's currently impossible to know whether an extension (or plugin with an active extensions flag) is part of the current build. Without even considering this, the current extension-scanning process simply attempts to load the extension (or plugin) out of the repository.

One potential solution for this problem would involve deferring extension scanning (or repeating it, with suppression of failures the first time around) until after project A's build completes. Then, load the extensions for project B just before attempting to build it. This should allow Maven to find A among the projects in the reactor, and use the newly minted artifact from that build as an extension in B. This approach uses the natural strengths of the project-sorter used to determine the build order under normal circumstances, and should allow Maven to give preference to artifacts from the reactor even when they also exist in the repository. The deferral approach would probably require us to build up a tree of project coordinates in the current build (which we should be able to do through the model-traversal logic in the extension-scanning process), then sort that tree into a graph of interdependencies before loading any extensions at all. Then, for any declared extensions or extension-plugins that are in the graph, we would defer loading that extension until just prior to building the project that uses it.

One important drawback with this approach involves profile activation. In order to build up a full list of project coordinates for the current build, we may need some extensions to be loaded in order to gain access to modules defined in profiles (so we can add them to the project-coordinate list and then recurse to their own modules). This creates a delicate balancing act of when to load different parts of the project information, in order to gain access to extensions as early in the process as possible.

Aside from the potential for chicken-and-egg situations, deferred extension loading would also mean that extension loading would be a two-stage process. First is the existing pre-scanning process, where most extensions will be identified and loaded. Following this is a second round of extension loading just before the build for that project begins, since it's assumed that project-sorting will have demanded that the extension be built ahead of its dependent.

Also, it's not clear how aggregator plugins might derail this logic, since they need access to all project instances when they execute. Another drawback of this approach is that it would negate a lot of the fail-fast behavior built into the current Maven, by deferring extension loading - and by extension, project instantiation for B - until after other projects have built...and some project builds are very time consuming.

[IMPORTANT] Modules defined in POM profiles are not scanned for extensions

In cases where profiles are used to change the behavior of builds on the broadest scales (the ServiceMix around SVN revId: 642243 is a great example of this), profiles are often used to switch between two completely separate sets of modules, with no modules defined in the root of the POM itself. When applying the current 2.1 extension-loading design to this scenario, none of the modules are currently scanned for extensions or plugins with an active extensions flag.

A potential solution would be to activate what profiles we can in the model before grabbing the modules-list to continue the extension-scanning process. I say "what profiles we can" because it's possible that custom profile activators may be used in some of the profiles; in this case, we must take extra care to either:

  1. avoid exceptions when these custom activators are used, but haven't yet been loaded (in case we cannot load them first)
  2. or, better yet: make sure any custom profile activators used by the current POM are loaded before the profiles are activated, to ensure all intended profile activation can take place.

[IMPORTANT] Plugins defined in POM profiles with an active extensions flag will not be loaded as extensions

This situation is very similar to that outlined in the section above, where the module-list scanned for extensions is incorrect due to some modules being defined in profiles. Likewise, plugins with extensions == true could be defined in profiles, and currently Maven would fail to find them during the extension-scanning process. Fixes for this will likely be the same as the above.

[IMPORTANT] Allow extension authors to specify which classes should be imported into project realms

In cases where the extension classpath is 2 or fewer artifacts, or the extension provides a lifecycle mapping, the current extension-loading mechanism in 2.1.x breaks backward compatibility. In such cases, 2.0.x loads the entire extension classpath is exposed to the Maven runtime for use in plugins or the core itself. By limiting class imports into the project realm to only component-implementation classes, Maven 2.1.x currently eliminates the possibility of accessing a component by a custom interface defined in the extension (through casting). It also prohibits access to components defined in dependencies of the extension artifact, since these dependency artifacts are not scanned for component definitions. Finally, anything else, such as custom datatypes, will also be isolated behind the extension-realm barrier.

In certain cases, extension developers need to provide more access to their components than just the interface from the core realm and the implementation class itself. This suggests something similar to OSGi, that would allow developers to specify a manifest of the classes exported, along with dependencies used, by the extension. Perhaps a bundling mechanism could provide these semantics. In any case, this is an issue that will be expanded in a separate proposal: Maven 2.1 Extension Bundling.