flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Re: Using native library in Flink
Date Tue, 18 Jul 2017 15:02:50 GMT
Hi Mike,

do you run Flink locally or in a cluster? You have to make sure that VM 
argument -Djava.library.path is set for all Flink JVMs. Job Manager and 
Task Managers might run in separate JVMs. Make also sure that the 
library is accessible from all node. I don't know what happens if the 
file is accessed by multiple processes/threads at the same time. It 
might also important where you put the static { ... } loading. It should 
be in the Function, because these classes get deserialized on the 
TaskManager.

I hope this helps.

Timo


Am 17.07.17 um 21:30 schrieb Mike Accola:
> I am new Flink user just trying to learn a little bit.  I am trying to
> incorporate an existing C++ library into a new Flink application.  I am
> seeing some strange behavior when trying to link in the native (C++)
> library using java via JNI.
>   
> I am running this on Linux (RHEL6)
>   
> I can run my application once without error.  Sometimes it will run
> successfully a 2nd or 3rd time.  However, eventually on a subsequent run,
> I get an exception about the the native library not being found:
>   
> java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
>          at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
>          at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>          at java.lang.System.loadLibrary(System.java:1122)
>          at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:10)
>   
> For debugging purposes for now, my native library does not have any
> external references.  It really contains 1 method that essentially does
> nothing.
>   
> The behavior seems to indicate that there is some kind of cleanup being
> done that "unloads" the native library.  I suspect this is somehow related
> to Flink's implementation of its library cache manager, but I have not
> been able to prove this yet.
>   
> A few more details:
>   
> - I have a c++ library libdummy2native.so that contains a method that can
> be invoked via JNI.
> - I have a jar containing a class, called Dummy2.  The Dummy2 constructor
> will invoke the JNI method.
> - The libdummy2native.so library is invoked with System.loadLibrary() like
> this:
>       static {System.loadLibrary("dummy2native"); }
> - In my simple Flink application, I have extended the ProcessFunction
> class.  Within this class, I have overriden processElement method that
> declares a Dummy2 object.
> - The Dummy2 class can be called and invoked without error when used in a
> standalone java program.
>   
> Any thoughts or ideas on what to try next would be appreciated. Initially,
> I'd be happy to be able to just explain this behavior.  I will worry about
> fixing it afterwards.
>   
> Thanks.
>
>
>
>


Mime
View raw message