flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eron Wright <eronwri...@gmail.com>
Subject Re: Using native library in Flink
Date Tue, 18 Jul 2017 21:39:52 GMT
The solution mentioned by Timo works well with a standalone Flink cluster
but might not work with a YARN or Mesos cluster.  An alternative is to have
your Java library contain the native library within itself, and to extract
it to a temporary directory before calling `System.loadLibrary(...)`.
Note that you lose the advantages of using the native OS's packaging system
(e.g. security patches, dependency management).   The TensorFlow Java
library demonstrates the technique:

https://github.com/tensorflow/tensorflow/blob/v1.2.1/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java

-Eron

On Tue, Jul 18, 2017 at 8:02 AM, Timo Walther <twalthr@apache.org> wrote:

> Hi Mike,
>
> do you run Flink locally or in a cluster? You have to make sure that VM
> argument -Djava.library.path is set for all Flink JVMs. Job Manager and
> Task Managers might run in separate JVMs. Make also sure that the library
> is accessible from all node. I don't know what happens if the file is
> accessed by multiple processes/threads at the same time. It might also
> important where you put the static { ... } loading. It should be in the
> Function, because these classes get deserialized on the TaskManager.
>
> I hope this helps.
>
> Timo
>
>
> Am 17.07.17 um 21:30 schrieb Mike Accola:
>
> I am new Flink user just trying to learn a little bit.  I am trying to
>> incorporate an existing C++ library into a new Flink application.  I am
>> seeing some strange behavior when trying to link in the native (C++)
>> library using java via JNI.
>>   I am running this on Linux (RHEL6)
>>   I can run my application once without error.  Sometimes it will run
>> successfully a 2nd or 3rd time.  However, eventually on a subsequent run,
>> I get an exception about the the native library not being found:
>>   java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
>>          at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
>>          at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>>          at java.lang.System.loadLibrary(System.java:1122)
>>          at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:
>> 10)
>>   For debugging purposes for now, my native library does not have any
>> external references.  It really contains 1 method that essentially does
>> nothing.
>>   The behavior seems to indicate that there is some kind of cleanup being
>> done that "unloads" the native library.  I suspect this is somehow related
>> to Flink's implementation of its library cache manager, but I have not
>> been able to prove this yet.
>>   A few more details:
>>   - I have a c++ library libdummy2native.so that contains a method that
>> can
>> be invoked via JNI.
>> - I have a jar containing a class, called Dummy2.  The Dummy2 constructor
>> will invoke the JNI method.
>> - The libdummy2native.so library is invoked with System.loadLibrary() like
>> this:
>>       static {System.loadLibrary("dummy2native"); }
>> - In my simple Flink application, I have extended the ProcessFunction
>> class.  Within this class, I have overriden processElement method that
>> declares a Dummy2 object.
>> - The Dummy2 class can be called and invoked without error when used in a
>> standalone java program.
>>   Any thoughts or ideas on what to try next would be appreciated.
>> Initially,
>> I'd be happy to be able to just explain this behavior.  I will worry about
>> fixing it afterwards.
>>   Thanks.
>>
>>
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message