flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Zagrebin <and...@ververica.com>
Subject Re: Could not load the native RocksDB library
Date Wed, 03 Jul 2019 15:08:40 GMT
Hi Samya,

Additionally to Haibo's answer:
Have you tried the previous 1.7 version of Flink? The Rocksdb version was
upgraded in 1.8 version.

Best,
Andrey

On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun <sunhaibotb@163.com> wrote:

> Hi,  Samya.Patro
>
> I guess this may be a setup problem. What OS and what version of JDK do
> you use?  You can try upgrading JDK to see if the issue can be solved.
>
> Best,
> Haibo
>
> At 2019-07-02 17:16:59, "Patro, Samya" <Samya.Patro@gs.com> wrote:
>
> Hello,
> I am using rocksdb for storing state . But when I run the pipeline I get
> the error   ”Could not load the native RocksDB library” .  Kindly can you
> check the configs and error stacktrace and suggest what am I doing wrong .
>
>
>
> Flink version  - 1.8.0
>
>
>
> <*dependency*>
>     <*groupId*>org.apache.flink</*groupId*>
>     <*artifactId*>flink-statebackend-rocksdb_2.11</*artifactId*>
>     <*version*>1.8.0</*version*>
> </*dependency*>
>
>
>
> This is  the flink checkpointing config I have used
>
>
>
> *executionEnvironment**.enableCheckpointing(**300000*
>
> *);
> executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
> executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(*
> *50000**);*
>
> *executionEnvironment.getCheckpointConfig().setCheckpointTimeout(**600000*
> *);
> executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(*
> *1**);*
>
> *executionEnvironment*
> *.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.*
> *RETAIN_ON_CANCELLATION**);*
>
> *StateBackend rocksDbBackend = **new **RocksDBStateBackend(parameter.get(*
> *"stateBackendPath"**),**true*
> *); **executionEnvironment**.setStateBackend(rocksDbBackend);*
>
>
>
> *When I run the pipeline, I get this error*
>
>
>
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
>
>     at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)
>
>     at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)
>
>     at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>
>     at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>
>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>
>     at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from
> any of the 1 provided restore options.
>
>     at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
>
>     at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)
>
>     at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)
>
>     ... 5 more
>
> Caused by: java.io.IOException: Could not load the native RocksDB library
>
>     at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)
>
>     at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)
>
>     at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)
>
>     at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
>
>     at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
>
>     ... 7 more
>
> Caused by: java.lang.UnsatisfiedLinkError:
> /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so:
> /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so:
> undefined symbol: malloc_stats_print
>
>     at java.lang.ClassLoader$NativeLibrary.load(Native Method)
>
>     at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
>
>     at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
>
>     at java.lang.Runtime.load0(Runtime.java:809)
>
>     at java.lang.System.load(System.java:1086)
>
>     at
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
>
>     at
> org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
>
>     at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)
>
>     ... 11 more
>
>
>
>
>
>
>
> *Thanks and  Regards, Samya Ranjan Patro Goldman sachs*
>
>
>
> ------------------------------
>
> Your Personal Data: We may collect and process information about you that
> may be subject to data protection laws. For more information about how we
> use and disclose your personal data, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to: www.gs.com/privacy-notices
>
>

Mime
View raw message