How to burn your weekend, with jackson-mapper

TL;DR - version 1.5.3 in pom.xml was wrong; changing to 1.9.10 fixed it.

We are still knee-deep past our project deadlines. There we were, generating the EAR file to be deployed on to our HSM-laden servers in office. Introduce my version. I'd been mucking around happily developing my own stuff. Now comes my turn to deploy my version merged from CVS into the JBoss. I'll just cut it short to say that, other than my version, nobody else could get a specific ajax function to operate properly.

The team became stuck for a day or two. Washing my hands off of it, "it was working when I left", and several hours later, they still couldn't get the clean CVS copy to do the job. None of their EAR files could. Then Saturday late night comes, and a colleague decides to head back to office after dinner. I joined her remotely from home. She mucked around for a good 2 hours. Still no dice. Then she decided to drop my webapp WAR into her EAR. It works! She diff'd the archive contents in BeyondCompare, and found several differences. We tested out several theories, but to no avail. It was around 2.30am when we pretty much decided to give it a rest.

Before she left, she noted one difference for me to investigate, since I was planning to head down this morning. The /lib files contained 2 different pairs of JAR files. Both jackson-mapper and jackson-core JARs represented themselves differently in her WAR and mine.

After researching, it turns out jackson-mapper has a dependency on jackson-core, meaning they should always come in pairs. I had both pairs of version 1.5.3 and 1.9.10 in my WAR.

Come morning, I was in the office testing out this newfound theory. In order not to let my miraculous EAR interfere, I only meddled with a copy of the EAR left behind for me by my colleague.

1. Control test - deploy her original, untainted EAR; true to form, the ajax call is broken.
2. Replace both jackson-mapper and jackson-core with v1.9.10 inside WAR; hurray, it works!
3. Additionally, due to extra instructions with another colleague, remove jackson-core and leaving solely jackson-mapper v1.9.10 within; expectedly, it doesn't work. His impression was our pom only indicating the use of the mapper JAR and not the core JAR, so maven truly did its job behind the scenes by including the required dependency without us even realising.

After some hemming and hawing because this guy is reluctant to upgrade from 1.5.3 to 1.9.10, in case it breaks anything else, I barely persuaded him to go ahead with it anyway or it'd remain a showstopper.

Our pom was indicating 1.5.3 all this while, because nobody wants to rock the boat with a new version since it works. Even though v1.5.3 is 4 years old and 1.9.10 is 2 years old at best. The version number in our pom for jackson-mapper was updated, so that the maven on everybody's machines would know to use the newer version instead.

Was it that hard to identify the problem? The server never provided any feedback in our logs. The ajax function obviously did its job, since the database was updated because of it. But with v1.5.3, the result never returned to the browser. The culprit could either be due to the @ResponseBody implementation being unavailable in 1.5.3 (unlikely), or the fact that the ajax call was using the jQuery implementation on a POST method instead of GET. We don't have time to point any more fingers.

The problem with the multitudes of libraries and frameworks available, is that through such obfuscation, identifying problems like this can become increasing difficult. And not forgetting the convenience of the Spring framework that performs autowiring with annotations. Those files compile fine and are happily packaged into WARs, JARs and EARs. Except they'd quietly fail at runtime.

Our project continues on its way with our testers picking out the bugs.

Loading DLL in a web-based applet

This is going to be lengthy and technical.

The problem
My project has a page that calls an applet that in turn loads a DLL to perform certain tasks. This works when the page is first loaded, but dies when the page refreshes itself. Log traces indicate that the applet dies when calling the System.loadLibrary with the DLL after the first time. Suspicions rose at the inner workings of either the applet jar in browser cache, garbage collection, or DLL being loaded. My debug research went both ways.

Caching applet.jar in browser
The search started with me locating the various mechanisms used to flag to the browser that the applet is not to be cached. I found out about cache_option and cache_archive pair. I revisited the legacy_lifecycle switch. I'd even attempted to switch over to the "recommended" method of using deployJava.js in place of my <applet> tag. Fortunately and unfortunately, I found out that the jsession variable was already appending to the applet.jar?jsession= due to the framework in place, so that was a dead-end. It still didn't work.

Garbage Collection
Attempts were made to explicitly call System.gc() in the applet stop() and destroy() methods. The class for loading the DLL was set to null prior. Doesn't work.

System.loadLibrary calling of the DLL
Turning my attention to the DLL loading mechanism, I ventured to unload the DLL. The standard pattern was to make use of a static call to System.loadLibrary in a Java class. What could possibly have gone wrong? I found out that people suggested to unload the DLL because they have a need for it. Following their words, I crafted my custom ClassLoader.

My project was a web application. The applet is stored within a jar, hosted on the web server. The custom ClassLoader has to be able to access the http:// URI scheme, read the jar file and dump out the class file from within. I improvised by combining the solutions from here, here and here.

MyCustomClassLoader:
public class MyCustomClassLoader extends ClassLoader {
    private String MyCustomAppletUrl;
    /**
     * The HashMap where the classes will be cached
     */
    private Map<String, Class<?>> classes = new HashMap<String, Class<?>>();

    public MyCustomClassLoader(String url) {
        this.MyCustomAppletUrl = url;
    }

    @Override
    public String toString() {
        return MyCustomClassLoader.class.getName();
    }

    @Override
    protected Class<?> findClass(String name) throws ClassNotFoundException {

        if (classes.containsKey(name)) {
            return classes.get(name);
        }

        byte[] classData;

        try {
            classData = loadClassData(name);
        } catch (IOException | URISyntaxException e) {
            throw new ClassNotFoundException("Class [" + name + "] could not be found at " + new File(name).getAbsolutePath(), e);
        }

        Class<?> c = defineClass(name, classData, 0, classData.length);
        resolveClass(c);
        classes.put(name, c);

        return c;
    }
   
    private byte[] loadClassData(String name) throws IOException, URISyntaxException {
        URL url = new URL(MyCustomAppletUrl);
        BufferedInputStream in = new BufferedInputStream(url.openStream());
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        int i;

        while ((i = in.read()) != -1) {
            out.write(i);
        }

        in.close();
        byte[] jarBytes = out.toByteArray();
        out.close();
       
        JarInputStream jis = new JarInputStream(new ByteArrayInputStream(jarBytes));
        JarEntry entry;
        InputStream inStream = null;
        while ((entry = jis.getNextJarEntry()) != null) {
            if (entry.getName().equals(name.replace(".", "/") + ".class")) {
                inStream = jis;
                break;
            }
        }
       
        if(inStream != null) {
            ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
            int nextValue = inStream.read(); 
            while (-1 != nextValue) { 
                byteStream.write(nextValue); 
                nextValue = inStream.read(); 
            }
           
            byte[] classData = byteStream.toByteArray();
            out.close();
            return classData;
        }
        return null;
    }
}

MyCustomLibraryLoader:

public class MyCustomLibraryLoader {
    private Class<?> MyCustomClass;
    private Object MyCustomClassObj;
   
    public MyCustomLibraryLoader(String url) throws ClassNotFoundException, InstantiationException, IllegalAccessException {
        MyCustomClassLoader cl = new MyCustomClassLoader(url);
        MyCustomClass = cl.findClass("com.nec.asia.MyCustom.client.MyCustom_Client");
        MyCustomClassObj = MyCustomClass.newInstance();
    }
   
    public String MyCustomClientAction(String inputXML) {
        System.out.println("MyCustomClientAction:");
        Method m;
        try {
            m = MyCustomClass.getMethod("MyCustom_Client_Action", String.class);
            return (String) m.invoke(MyCustomClassObj, inputXML);
        } catch (Exception e) {
            e.printStackTrace();
        }
       
        return null;
    }
   
    public String MyCustomVerifyView(String inputXML) {
        System.out.println("MyCustomVerifyView:");
        Method m;
        try {
            m = MyCustomClass.getMethod("MyCustom_VerifyView", String.class);
            return (String) m.invoke(MyCustomClassObj, inputXML);
        } catch (Exception e) {
            e.printStackTrace();
        }
       
        return null;
    }
}

And guess what? It didn't work. I meant that I managed to use them to retrieve the remote jar, extract the class and load the DLL, but my original problem remained. Reviewing another product with a similar setup proved vital, and had me deleting the above files.

The solution
Yes I had it working in the end. The class file with the native method calls was placed in a separate jar all by its lonely little self. This second jar was then added to the archive param in the applet:
<param name="archive" value="mywebapp-applet.jar,lonely-dll-caller.jar" />

And the applet behaved like my drama was totally uncalled for. Other stuff I decided to keep for now:

• the legacy_lifecycle parameter
• use of <applet> instead of <object> or even use deployJava.js
• don't forget to include 
• <param name="type" value="application/x-java-applet;" />
• <param name="scriptable" value="true" />
• <param name="mayscript" value="true" />

That's all folks! The applet loads the DLL process properly now, even after subsequent reloading of the same page.

20 steps to Java/DLL integration over JNI

I'm not well-versed in C++ nor .Net, so I was still learning as I encountered this at work. While not being overly generic yet not too specific about the ordeal I encountered, it involved getting a DLL to work with our Java components. Sounds straightforward enough? It was... kind of. Guy 1 is in charge of developing the main DLL and its compilation stuff. Guy 2 is versed in C++ but understands Java sufficiently to provide the DLL for the Java Native Interface (JNI) adapter to the main DLL from Guy 1, along with basic testing classes in Java.

It certainly looked like he had already done all my work for me. Not quite. I'd have to proceed with his setup to get their bits working with the rest of our web application in a Java framework. Here's how it went... (WARNING: wall of text that includes technical stuff and bitching)

TL;DR - integrated Java with DLL using JNI, shenanigans with co-workers ensue, Java calls to DLL success.

Step 1: Setup the DLL paths and environment provided by Guy 2 in my Eclipse,

Step 2: Realise Guy 2 provided basic XML parsing via DocumentBuilder,

Step 3: Convert everything he's done into XStream-friendly POJO,

Step 4: Realise XStream is an external library and commence search for alternative to reduce additional footprint,

Step 5: Took the advice of Blaise Doughan and adopted Java Architecture for XML Binding (JAXB) because it's part of the javax.xml.bind package,

Step 6: Discovered the joys of source annotations and XML adapters,

Step 7: Realise that the XML definition that was long decided before I was brought on board (and already implemented on the .Net server side) is complicated (e.g.:

<fans>
  <fan type="standing"/>
  <fan type="desk"/>
  <fan type="celebrity"/>
</fans>

) by attribute naming that prevented JAXB from unmarshalling the String to object,

Step 8: Came up with my own convoluted (I swear it's as elegant as I could get it) solution to first read the <fans> to determine what types were in the list provided, before manually stripping <fans> from the String front and back and running it through JAXB another time, (if you have a better way, let me know!)

Step 9: Attempt to run the whole shebang with the DLL,

Step 10: Act surprised that JNI complained of an UnsatisfiedLinkError for the DLL,

Step 11: Figured that Eclipse had to be told where to look with -Djava.library.path="C:/that_dumb_place/Dlls" when running my test program,

Step 12: Nope still won't work, spend the morning poking around some more before giving up and asking Guy 2 if I'd missed out anything,

Step 13: Guy 2 refers me to Guy 1 for the DLL portion, "fair enough" I lied to myself,

Step 14: Decided I should test further without the Java components by first running the .exe test program they provided... crashed,

Step 15: Approach Guy 1 who tells me he'd investigate the error on his end first, while I proceed to lunch,

Step 16: Guy 1 returns to offer me his solution of running his test program on his spare laptop that's been configured to work bug-free, instead of my problematic laptop that's long been tainted with other muck from previous projects,

Step 17: "Does this thing have Java installed?", I queried. Guy 1immediately forbids me from running my codes on his precious laptop. Explained to him I needed to run my codes with his because I had to integrate them for this project. Seeing there was no other way to squirm out of this, he promptly took over my laptop to figure out how to get his test program running,

Step 18: I twirled around in my chair for almost 2 hours before he returned triumphant. It worked! At last, his test program works on my laptop. He had installed a new driver that goes along with a new model of the Human-Interface Device (HID) that we have already provided the client.

Step 19: Indeed, System.load("That_Darn_JNI") works with the DLL from Guy 2 in his codes that I've since adopted and turned it from a baby into an adolescent, complete with full-blown set of beans and action classes, just as long Eclipse knows about that -Djava.library.path it cleverly denies knowing about,

Step 20: Try running my test program. Nope, Unsatisfied remains. Made comparison between JNI DLL and existing known working DLL from another product, and finally getting around to rectify minor bugs in my codes to marshal/unmarshal between the XML and POJO.

So I finally got what I wanted... well almost. The following addendum is the cherry missing from my cake...

Step 21: Realised that because I'd shifted the DLL away from the original XYZ.Product package into my Java-convention style of com.XYZ.asia.product.client caused the JNI to not map to the native DLL, and that it'd be confusing to leave the product package as XYZ.Product while the rest of the project is differently identified,

Step 22: Ask Guy 2 about the JNI, he patiently explained about how the header works but ultimately suggested that I should talk to Guy 1,

Step 23: Locate Guy 1 about passing him the JNI header to generate the DLL for me. Nope, his bunch of codes would take him way longer to generate the resulting DLL,

Step 24: Return to Guy 2 about it. He relents, and I generate the JNI header via javah.exe mapping to the relocated JNI wrapper

Step 25: I have yet to test this new DLL yet, although I reckon it should function as expected... and hopefully this tiring liaison with them is over for the time being...

And thus, at the end of the day, I got my DLL the way I wanted it, the HID works on my machine, the client-server calls are working and I'm getting the XML strings parsed correctly into Java objects. It's been gruelling, but educational.

To be continued...