hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kihwal Lee <kih...@yahoo-inc.com>
Subject Re: MPI: Java/JNI help
Date Tue, 31 Jan 2012 23:35:25 GMT
There might be other tricks you can play with CL, but here is my idea: You could have the initial
jni native lib to become a sort of wrapper to dlopen() the real thing (the one plug-ins depend
on) with RTLD_GLOBAL, so that the fact that the jni library is loaded in a specific name space
does not matter.


On 1/31/12 4:34 PM, "Ralph Castain" <rhc@open-mpi.org> wrote:

I was able to dig further into this, and we believe we finally tracked this down to root cause.
It appears that java loads things into a private, as opposed to global, namespace. Thus, the
Java MPI bindings load the initial libmpi just fine.

However, when libmpi then attempts to load the individual plug-ins beneath it, the load fails
due to "unfound" symbols. Our plug-ins are implemented as individual dll's, and reference
symbols from within the larger libmpi above them. In order to find those symbols, the libraries
must be in the global namespace.

We have a workaround - namely, to disable dlopen so all the plug-ins get pulled up into libmpi.
However, this eliminates the ability for a vendor to distribute a binary, proprietary plug-in
that we "absorb" during dlopen. For the moment, this isn't a big deal, but it could be an
issue down the line. We have some ideas on how to resolve it internally, but it would take
a fair amount of work, and have some side-effects.

Does anyone know if it is possible to convince java to use the global namespace? Or can you
point me to someone/someplace where I should explore the question?


On Jan 30, 2012, at 5:13 PM, Kihwal Lee wrote:

> It doesn't have to be static.
> Do architectures match between the node manager jvm and the library?
> If one is 32 bit and the other is 64, it won't work.
> Kihwal
> On 1/30/12 5:58 PM, "Ralph Castain" <rhc@open-mpi.org> wrote:
> Hi folks
> As per earlier emails, I'm just about ready to release the Java MPI bindings. I have
one remaining issue and would appreciate some help.
> We typically build OpenMPI dynamically. For the Java bindings, this means that the JNI
code underlying the Java binding must dynamically load OMPI plug-ins. Everything works fine
on Mac. However, on Linux, I am getting dynamic library load errors.
> I have tried setting -Djava.library.path and LD_LIBRARY_PATH to the correct locations.
In both cases, I get errors from the JNI code indicating that it was unable to open the specified
dynamic library.
> I have heard from one person that JNI may need to be built statically, and I suppose
it is possible that Apple's customized Java implementation specifically resolved that problem.
However, all the online documentation I can find indicates that Java on Linux should also
be able to load dynamic libraries - but JNI is not specifically addressed.
> Can any of you Java experts provide advice on this behavior? I'd like to get these bindings
> Thanks
> Ralph

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message