Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

Using Maven2 to build projects which use JNI code

This page documents my experiences with supporting projects which use JNI code, and describes the solution I developed and the tools I use, in the hope that it may save somebody else this pain.

Background

I work in telecoms, which means that we have quite a lot of projects which have a greater or lesser quantity of native code, either because they interface to the operating system at a low level, or simply as a way of dealing with the real-time requirements of our software. As time has gone by, we've built up a reasonably complex heirarchy of applications with native code which depend on libraries with native code, and some of the libraries even export native-level symbols to application native code.

Our investigations with converting JNI-free projects to a Maven2 build process were extremely positive, and it therefore soon became desirable to convert all of our projects, including those with JNI requirements, to Maven2. This led to the following requirements for the build process.

Requirements

  1. The build/release process should match as closely as possible that of a non-JNI project - checkout followed by 'mvn package', etc.
  2. As this functionality will be common to many projects, long incantations of plugin configuration in each pom are unacceptable; it must be possible to either factor everything out to a common parent pom, or just to have sensible defaults which build everything. For example, it should be possible to make use of a library using JNI code - and have it work for unit tests, assemblies, etc - just by adding it to your <dependencies/> element.
  3. It should be possible to run an application with as little extraneous scripting as possible. Essentially, this means: unzip assembly; run jar -jar on application jar.
  4. It should be possible to call functions in one JNI library from another. Typically this means having the library's include files and dll available at compile time, and having the library available for dynamic linking at runtime.
  5. The build process must be portable from one platform to another. I'm happy to require that building for Windows requires Cygwin, but it should be possible to have builds for different platforms alongside each other in the repository, and the right build be chosen when building.
  6. Our legacy build process must continue to work until the whole company can be migrated to maven2.

For bonus points:

  1. During the development cycle, it shouldn't be necessary to run the entire maven build cycle for each change. The user's IDE can be configured to build the Java side, so we need a means of quickly building the native side. This also offers an easier upgrade path from our legacy build system, fwiw.

Possible solutions

There are obviously many ways to skin this particular cat; I'll discuss a few of the options.

Native-maven-plugin

The first place to look is obviously to see if anybody else has solved this problem. I therefore made a start with the native-maven-plugin. This, however, has a number of problems, which means that it totally fails to meet my requirements.

  • The showstopper is that the native code ant the java code are separate projects. This means that it's impossible to get things to build in the right order, because we have to do the following.
    1. compile java source to class files
    2. run javah on class files
    3. compile and link native code
    4. run unit tests for java code

Some messing about with a very complex project structure might be possible, but it's certain to be ugly and very hard to set up for each of many projects.

  • If you build a library which has a JNI component, making use of it is very complex. Essentially, maven2 downloads the jni library to the depths of your local repository, where it then can't be found by a System.load().
  • Again, when you depend on a library with a JNI component, you need complex incantations in your pom, to depend on both the jar, and a separate profile for each target platform to depend on the native library.
  • Building the native code always requires running the full maven build.

JNI library as an attached artifact

The next possibility considered was to build the native library as an 'attached artifact' - in much the same way as a javadoc or source jar can be attached to the main artifact. This solved some of the problems, especially the problem with build order; depending on a library with jni code was still a nightmare, however.

JNI library archived within the jar

The solution I ended up using was to store the compiled jni library in the jar alongside the class files.

This means either cross-compiling for all possible architectures, or having a different jar, more simply, having a different jar for each architecture. This latter fits quite well with our setup - where almost all of our machines are Linux-i386, with a smattering of win32 boxes.

Sadly System.load() can't cope with loading libraries from within a jar, so we'll therefore need a custom loader which extracts the library to a temporary file at runtime; this is obviously achievable, however.

Implementation

For an example of a project using this implementation, you may like to download simple-native-example. That directory contains source archives (eg simple-native-example-1.0.0-src.tar.bz2), as well as binaries compiled for Linux-i386, and javadocs, and may provide helpful examples of the concepts below.

Directory structure

We need a place to put native code. This should naturally be src/main/native in the standard directory layout; for a more complex project it may be appropriate to further split this into src and include.

The JNI library should be built straight into a subdirectory of outputDirectory; by doing so, it will automatically get built into the project jar, as well as being in the right place when running the project from an IDE which just uses the .class files straight out of outputDirectory. I decided to put such my jni libraries in META-INF/lib, but this is obviously not set in stone.

Building the JNI

The first thing we need to do (after creating the .java files, with suitable native methods, of course), is to build the JNI library. I chose to do this using a Makefile, and a maven-exec-plugin execution, as this makes it reusable between different build environments, and is easily run independently of Maven. It's also an easy way of making sure the right files, and only the right files, get recompiled.

The Makefile needs to:

  • use javah to create .h files from the .class files in the outputDirectory.
  • use gcc to compile c into object files
  • link the object files into the dynamic library.

We now need for make to be run after the .class files are built; I did this by attaching an execution to the process-classes phase:

Library loader

We now have our JNI library on the class path, so we need a way of loading it. I created a separate project which would extract JNI libraries from the class path, then load them. Find it at http://opensource.mxtelecom.com/maven/repo/com/wapmx/native/mx-native-loader/1.0.3. This is added as a dependency to the pom, obviously.

To use it, I simply call com.wapmx.nativeutils.jniloader.NativeLoader.loadLibrary(libname). More information is in the javadoc for NativeLoader.

I generally prefer to wrap such things in a try/catch block, as follows:

We should now be at the point where our junit tests work from maven; a mvn test should work! It should also work fine from an IDE.

Setting the classifer

As I mentioned earlier, we need to be able to keep builds for multiple architectures alongside one another in the repository, so we need to distinguish between different builds somehow. To make this work, I created a plugin which sets a classifier on the project jar. Basically, that means we add -${os.name} to the name of the jar, and when we refer to it in another project's dependencies, we add a <classifer>${os.name}</classifier> element.

The plugin is at http://opensource.mxtelecom.com/maven/repo/com/wapmx/maven2/mx-native-maven-plugin/1.0.4/; its documentation is at http://opensource.mxtelecom.com/maven/mx-native-maven-plugin/; and to use it, I put this in my project pom:

It's a terrible hack, but it does work quite well in practice.

Summary

It really is as simple as that. You should now be able to use your JNI-enabled project exactly as you would any other project - with the one proviso that, if you depend on it from another project, you must put a a <classifer/> element in the dependency. For example:

  • No labels