Using Maven2 to build projects which use JNI code
This page documents my experiences with supporting projects which use JNI code, and describes the solution I developed and the tools I use, in the hope that it may save somebody else this pain.
I work in telecoms, which means that we have quite a lot of projects which have a greater or lesser quantity of native code, either because they interface to the operating system at a low level, or simply as a way of dealing with the real-time requirements of our software. As time has gone by, we've built up a reasonably complex heirarchy of applications with native code which depend on libraries with native code, and some of the libraries even export native-level symbols to application native code.
Our investigations with converting JNI-free projects to a Maven2 build process were extremely positive, and it therefore soon became desirable to convert all of our projects, including those with JNI requirements, to Maven2. This led to the following requirements for the build process.
- The build/release process should match as closely as possible that of a non-JNI project - checkout followed by 'mvn package', etc.
- As this functionality will be common to many projects, long incantations of plugin configuration in each pom are unacceptable; it must be possible to either factor everything out to a common parent pom, or just to have sensible defaults which build everything. For example, it should be possible to make use of a library using JNI code - and have it work for unit tests, assemblies, etc - just by adding it to your
- It should be possible to run an application with as little extraneous scripting as possible. Essentially, this means: unzip assembly; run
jar -jaron application jar.
- It should be possible to call functions in one JNI library from another. Typically this means having the library's include files and dll available at compile time, and having the library available for dynamic linking at runtime.
- The build process must be portable from one platform to another. I'm happy to require that building for Windows requires Cygwin, but it should be possible to have builds for different platforms alongside each other in the repository, and the right build be chosen when building.
- Our legacy build process must continue to work until the whole company can be migrated to maven2.
For bonus points:
- During the development cycle, it shouldn't be necessary to run the entire maven build cycle for each change. The user's IDE can be configured to build the Java side, so we need a means of quickly building the native side. This also offers an easier upgrade path from our legacy build system, fwiw.
There are obviously many ways to skin this particular cat; I'll discuss a few of the options.
The first place to look is obviously to see if anybody else has solved this problem. I therefore made a start with the native-maven-plugin. This, however, has a number of problems, which means that it totally fails to meet my requirements.
- The showstopper is that the native code ant the java code are separate projects. This means that it's impossible to get things to build in the right order, because we have to do the following.
- compile java source to class files
- run javah on class files
- compile and link native code
- run unit tests for java code
Some messing about with a very complex project structure might be possible, but it's certain to be ugly and very hard to set up for each of many projects.
- If you build a library which has a JNI component, making use of it is very complex. Essentially, maven2 downloads the jni library to the depths of your local repository, where it then can't be found by a
- Again, when you depend on a library with a JNI component, you need complex incantations in your pom, to depend on both the jar, and a separate profile for each target platform to depend on the native library.
- Building the native code always requires running the full maven build.
JNI library as an attached artifact
The next possibility considered was to build the native library as an 'attached artifact' - in much the same way as a javadoc or source jar can be attached to the main artifact. This solved some of the problems, especially the problem with build order; depending on a library with jni code was still a nightmare, however.
JNI library archived within the jar
The solution I ended up using was to store the compiled jni library in the jar alongside the class files.
This means either cross-compiling for all possible architectures, or having a different jar, more simply, having a different jar for each architecture. This latter fits quite well with our setup - where almost all of our machines are Linux-i386, with a smattering of win32 boxes.
Sadly System.load() can't cope with loading libraries from within a jar, so we'll therefore need a custom loader which extracts the library to a temporary file at runtime; this is obviously achievable, however.