flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathi Chowdhury <Sathi.Chowdh...@elliemae.com>
Subject trying to externalize checkpoint to s3
Date Tue, 23 May 2017 01:34:13 GMT
We are  running flink 1.2 in pre production
I am trying to test checkpoint stored in external location in s3

I have set these below in flink-conf.yaml

state.backend: filesystem
state.checkpoints.dir: s3://abc-checkpoint
state.backend.fs.checkpointdir: s3://abc-checkpoint

I get this failure in job manager log
java.lang.Exception: Cannot initialize File System State Backend with URI 's3://abc-checkpoint.
        at org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:57)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBackend(StreamTask.java:719)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:223)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Cannot use the root directory for checkpoints.
        at org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:225)
        at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:153)
        at org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.createFromConfig(FsStateBackendFactory.java:5
Any clue? I thought as I am using EMR Hadoop to s3 integration is already working.
=============Notice to Recipient: This e-mail transmission, and any documents, files or previous
e-mail messages attached to it may contain information that is confidential or legally privileged,
and intended for the use of the individual or entity named above. If you are not the intended
recipient, or a person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure, copying, printing,
distribution or use of any of the information contained in or attached to this transmission
is STRICTLY PROHIBITED. If you have received this transmission in error, please immediately
notify the sender by telephone or return e-mail and delete the original transmission and its
attachments without reading or saving in any manner. Thank you. =============
View raw message