OSGi Robots

JNI, Maven and OSGi

2009-04-25T00:42:00.011+10:00

I've done my best to avoid JNI like the plague until now. Usually I would pick one side of the fence and code in either pure C or pure Java. This is my first attempt to do both, and after quite a steep learning curve it doesn't seem any more attractive. However it could certainly be worse. Thanks to a fantastic and seemingly magical build tool, Maven, and a couple of well configured plugins, I was able to create an OSGi compliant bundle including a native library wrapper for the Linux Robotics Framework.

The goal of this whole excercise is to produce a single jar file which contains a native library, can be deployed in an OSGi framework, and ofcourse work when other bundles try to access methods with native hooks.

Those of you that have dabbled with JNI, and everyone else who writes any native code will know that to load a native library requires the binaries to be visible to the OS and not hidden away inside a jar. I was very happy to discover that an OSGi framework is nice enough to unpack native libraries to the file system for you, and then point the JVM to them when your bundle tries to load them.

OSGi bundles are usually expected to be portable and able to be deployed within any framework instance. Adding native libraries to a bundle does tie them to a particular OS and processor architecture. The OSGi specification does however, include a Bundle-NativeCode header which allows developers to notify the OSGi framework of native libraries included in the bundle, as well as the environments they require to run. Multiple native binaries can be included in a bundle, each with their own dependencies and the framework will pick and unpack to correct one. So we can recover some of our bundle's portability by including native libraries for different environments. If there is no binary provided for the environment in question, the bundle will gracefully refuse to load with the message:

org.osgi.framework.BundleException: Unable to select a native library clause.

This about as close as you can get to complete JVM portability when using any JNI.

To achieve this feat I'm using Maven with two explicit plugins. The freehep-nar-plugin will process your java source code with javah to produce some header files. Then it will compile you implementation of these headers and package them into a shared library. It will also create a pile of meta-data which comes in handy when you want to distribute things. Unfortunately, this plugin packages your java interface (jar) separately from the native libraries (nar). While this is a good thing and allows much more flexibility when managing these artifacts, it's not what we want when deploying to an OSGi framework.

The maven-bundle-plugin is very helpful when creating OSGi bundles. Although inserting manifest headers is not that difficult, it can be quite tedious and incredibly difficult to debug if you get it wrong. This plugin will process your compiled classes and calculate all the required import package headers, based on their dependencies. It will also verify your bundles manifest to ensure all is well.

Both of these plugins expect to create an artifact of their own type. To get the result needed, we need to get a bit creative. The final packaging we want is a bundle, so this is what we specify in our projects pom.xml. This means that the nar-plugin goals need to be mentioned explicitly. Add this between your nar-plugin tags.


<executions>
<execution>
  <goals>
    <goal>nar-javah</goal>
    <goal>nar-compile</goal>
    <goal>nar-system-generate</goal>
    <goal>nar-integration-test</goal>
  </goals>
</execution>
</executions>

That will generate the headers, compile the source code, create a convenient little class to load the library and run the integration test. It will not however, create the nar artifact, which is fine by me.

This will now give us a shared library built in our target directory, so now we need to include it in our bundle. Insert this between your build tags.


<resources>
<resource>
  <directory>target/nar</directory>
  <includes><include>lib/**</include></includes>
  <excludes><exclude>*.flag</exclude></excludes>
</resource>
</resources>

This will include all native libraries as resources in the final jar.

Finally we need to add meta-data to the jar so an OSGi framework knows what to do with it. Add this between your plugins tags.


<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
  <instructions>
    <Export-Package>com.osgibots.lrf4j</Export-Package>
    <Private-Package></Private-Package>
    <Bundle-Activator>com.osgibots.lrf4j.Activator</Bundle-Activator>
    <Bundle-NativeCode>
      lib/i386-Linux-g++/jni/liblrf4j-0.1-SNAPSHOT.so;
        osname=Linux;processor=x86
    </Bundle-NativeCode>
  </instructions>
</configuration>
</plugin>

Take note of the Bundle-NativeCode header. This specifies the native libraries available inside the bundle and which environment they are intended to run on. A list of canonical names is available within the OSGi specification.

The OSGi specification mentions the fact that "...only one class loader can load a native library as specified by an absolute path. Loading of a native library file by multiple class loaders (from multiple bundles, for example) will result in a linkage error." This is important to note as most implementations of an JNI involve static initializers to call System.loadLibrary(). If you've read my last post on static variables in OSGi, you'll know this will cause issues.

The easiest thing to do is to load the native library from your bundles activator.


public class Activator implements BundleActivator {

  public void start(BundleContext arg0) throws Exception {
      NarSystem.loadLibrary();
  }

  public void stop(BundleContext arg0) throws Exception {

  }
}

Note the usage of NarSytem.loadLibrary(). This helps us avoid changing constants within the source code with every new minor release.

I've skipped an awful lot of detail on the usage of these two plugins. You can get you hands on my example via my Mercurial repository:


hg clone http://hg.osgibots.com/lrf4j

This is an JNI wrapper for the Linux Robotics Framework which you will need available on your LD_LIBRARY_PATH if you hope to build the example. Of course you can just browse the code online.

There are other approaches to combining these two plugins. The ideas above work best when you are only planning to deploy to one environment, and therefore, only need to compile the native library once. If you want to include binaries for other environments, you will be better off creating two projects. One project to create the nar artifacts for various environments and deploy them to a repository. The second is to fetch all those artifacts and compile them into a single bundle. Unless you can cross compile all your target binaries on one machine, you'll have a great time keeping your code base in sync...

Static Variables and OSGi

2009-04-24T22:53:00.000+10:00

I've often run across discussion involving best-practice-zealots and their opinions on the use of static variables, often quite strongly against. I personally believe they have their place in the world and have used them often to great affect. However, when developing an application within an OSGi framework the game changes quite significantly.

Most developers take it for granted that a static variable will only have one instance within their application (per JVM). The fact is that a static variable will only have one instance per class-loader. After dealing with the millions of cryptic stack traces I found when first learning about OSGi, I expect everyone on the same path will soon realise that each bundle has its own class-loader. After all, it is written in the spec which I'm sure you've all read.

What this means is that some classes can be loaded by two distinct class loader instances within the same JVM. I've been unable to find a definitive answer regarding this behavior in various framework implementations and my knowledge of class loading is quite shallow. If anyone can shed some light, I would appreciate it. Having said that, if you've ever come a across an exception like

java.lang.ClassCastException: org.example.Foo cannot be cast to org.example.Foo

then you've witnessed the symptoms of uncoordinated class-loaders first hand. The number of times I saw this early on provides evidence to my claim.

This of course leads to some interesting errors, mostly from Singletons that have not been created completely. One major factor in an APIs OSGi compatibility (after a lack a manifest headers) is the use of static instances and how they behave in a multi-bundle, dynamic environment.

tweenbots

2009-04-12T12:58:00.000+10:00

Building a robot capable of traversing the streets of New York is no small task. The environment is unpredictable, the terrain is in no way smooth not to mention all the rules and etiquette of navigating a street without being hit by a car or arrested.

There is however, a technology that makes this task trivial and all it takes is a friendly face.

OSGi Bots

2009-04-06T16:23:00.000+10:00

I am in my final year of my Software Engineering degree at the Australian National University. As part of this final year I am spending one semester researching the use of OSGi technologies in distributed robotic systems. Having worked previously on the Linux Robotic Framework where I built 'Buzz' pictured to the left, I was fascinated by the use of software in real world machines, much more so than server and desktop applications.

I first came across OSGi when searching for a plugin/update management mechanism for an in-house software product I was developing at my day job. It appeared quite daunting at first, as it required a complete overhaul of the software architecture and introduced many new capabilities I had not considered. Thanks to some brilliant work by the developers at Dynamic Java I was able to create a modular, remotely updatable, and very maintainable desktop application without much fuss.

I immediately recognized the potential for OSGi and came up with a lot of scenarios where I'd like to see it used. One of these being a control system/platform for a distributed network of robotic agents. Others have incorporated OSGi into IDEs and application servers and several groups have dedicated themselves to supporting the OSGi movement.

It is difficult to explain the value of OSGi in a single paragraph so I strongly suggest you following the links and try to familiarize yourself with it, rather than taking my shallow explanation. An OSGi framework provides a dynamic and modular support system for Java. It addresses some of the shortcomings of bundle management within a JVM and greatly increases deployment capabilities. The image to the right has been taken form the OSGi specification version 4.1 and shows the basic architecture of an OSGi runtime. The execution environment will be a JavaSE virtual machine in most cases, but it can also be run on various realtime VMs, JavaME and even Dalvik. The bundles are the actual application packages.

My objective this semester is to demonstrate the use of OSGi in a robotic system. I hope to show that the dynamic capabilities of an OSGi based software system will allow for more powerful and creative configurations of individual robotic agents as well as complex systems of cooperating agents. Combing OSGi with other projects such as JXTA and R-OSGi will hopefully demonstrate a very powerful system of collaboration in dynamic systems.

I have started this blog to summarize my research and results in order to receive feedback from people much smarter than myself. I will be the first to admit I am in-experienced in robotics and quite naive with my expectations. Hopefully constructive criticism and feedback will help me reach a more realistic and practical outcome.