hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Castain <...@open-mpi.org>
Subject Re: MPI: Java/JNI help
Date Wed, 01 Feb 2012 16:37:24 GMT
Thanks Kihwal! This was an excellent suggestion and worked great!

Should have the Java bindings out shortly...

On Jan 31, 2012, at 4:35 PM, Kihwal Lee wrote:

> There might be other tricks you can play with CL, but here is my idea: You could have
the initial jni native lib to become a sort of wrapper to dlopen() the real thing (the one
plug-ins depend on) with RTLD_GLOBAL, so that the fact that the jni library is loaded in a
specific name space does not matter.
> 
> Kihwal
> 
> On 1/31/12 4:34 PM, "Ralph Castain" <rhc@open-mpi.org> wrote:
> 
> I was able to dig further into this, and we believe we finally tracked this down to root
cause. It appears that java loads things into a private, as opposed to global, namespace.
Thus, the Java MPI bindings load the initial libmpi just fine.
> 
> However, when libmpi then attempts to load the individual plug-ins beneath it, the load
fails due to "unfound" symbols. Our plug-ins are implemented as individual dll's, and reference
symbols from within the larger libmpi above them. In order to find those symbols, the libraries
must be in the global namespace.
> 
> We have a workaround - namely, to disable dlopen so all the plug-ins get pulled up into
libmpi. However, this eliminates the ability for a vendor to distribute a binary, proprietary
plug-in that we "absorb" during dlopen. For the moment, this isn't a big deal, but it could
be an issue down the line. We have some ideas on how to resolve it internally, but it would
take a fair amount of work, and have some side-effects.
> 
> Does anyone know if it is possible to convince java to use the global namespace? Or can
you point me to someone/someplace where I should explore the question?
> 
> Thanks
> Ralph
> 
> On Jan 30, 2012, at 5:13 PM, Kihwal Lee wrote:
> 
>> It doesn't have to be static.
>> Do architectures match between the node manager jvm and the library?
>> If one is 32 bit and the other is 64, it won't work.
>> 
>> Kihwal
>> 
>> On 1/30/12 5:58 PM, "Ralph Castain" <rhc@open-mpi.org> wrote:
>> 
>> Hi folks
>> 
>> As per earlier emails, I'm just about ready to release the Java MPI bindings. I have
one remaining issue and would appreciate some help.
>> 
>> We typically build OpenMPI dynamically. For the Java bindings, this means that the
JNI code underlying the Java binding must dynamically load OMPI plug-ins. Everything works
fine on Mac. However, on Linux, I am getting dynamic library load errors.
>> 
>> I have tried setting -Djava.library.path and LD_LIBRARY_PATH to the correct locations.
In both cases, I get errors from the JNI code indicating that it was unable to open the specified
dynamic library.
>> 
>> I have heard from one person that JNI may need to be built statically, and I suppose
it is possible that Apple's customized Java implementation specifically resolved that problem.
However, all the online documentation I can find indicates that Java on Linux should also
be able to load dynamic libraries - but JNI is not specifically addressed.
>> 
>> Can any of you Java experts provide advice on this behavior? I'd like to get these
bindings released!
>> 
>> Thanks
>> Ralph
>> 
>> 
> 
> 


Mime
View raw message