Hi,  Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use?  You can try upgrading JDK to see if the issue can be solved.

Best,
Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <Samya.Patro@gs.com> wrote:

Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error   ĦħCould not load the native RocksDB libraryĦħ .  Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

 

Flink version  - 1.8.0

 

<dependency>
    <
groupId>org.apache.flink</groupId>
    <
artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <
version>1.8.0</version>
</
dependency>

 

This is  the flink checkpointing config I have used

 

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(
50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(
1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

 

When I run the pipeline, I get this error

 

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

 

 

Thanks and  Regards,
Samya Ranjan Patro
Goldman sachs

 




Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices