flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siew Wai Yow <wai_...@hotmail.com>
Subject Re: Questions regarding to Flink 1.5.0 REST API change
Date Thu, 21 Jun 2018 06:07:58 GMT
Thanks Chesnay, the application will take value from "state.savepoints.dir" as default if set
target-directory to nul. But then it trying to create the directory in local machine, which
caused the below error because it is a HDFS directory. The same URL works in previous Flink
1.3.2. Is something break in Flink 1.5.0? Or anything we need to extra configure? Thank you.

Caused by: java.io.IOException: Mkdirs failed to create hdfs://
        at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:271)
        at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
        ... 25 more



state.backend: rocksdb
state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
state.backend.incremental: true
state.backend.async: true
state.checkpoints.num-retained: 5
state.savepoints.dir:  hdfs://


From: Chesnay Schepler <chesnay@apache.org>
Sent: Wednesday, June 20, 2018 3:15 PM
To: user@flink.apache.org
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

I think you can set the target-directory to null. But I'm not sure why this particular request
requires this, other request allow optional fields to simply be ommitted...

On 20.06.2018 06:12, Siew Wai Yow wrote:

Hi all,

Seems pass in target-directory is a must now for checkpoints REST API, and the status will
not response with save point directory anymore. I can pass in but the information is redundant
with the same already defined in flink-config.yaml. May I know is there a way to retrieve
the save point directory from flink-config.yaml in flink application? I am not able to get
it from env.getConfig(). Thank you.

From: Chesnay Schepler <chesnay@apache.org><mailto:chesnay@apache.org>
Sent: Tuesday, June 19, 2018 11:55 PM
To: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: Questions regarding to Flink 1.5.0 REST API change

1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" query parameter

2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a trigger id, used for 3).
"target-directory" : {

      "type" : "string"
    "cancel-job" : {
      "type" : "boolean"

3. GET to /jobs/:jobid/savepoints/:triggerid

On 19.06.2018 17:40, Esteban Serrano wrote:
For #1, you need to use a PATCH request to "/jobs/:jobid"

On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_yow@hotmail.com<mailto:wai_yow@hotmail.com>>


Regarding to Flink 1.5.0 REST API breaking change,

  *   The REST API to cancel a job was changed.
  *   The REST API to cancel a job with savepoint was changed.

I have few dump questions,

  1.  Any replacement for cancellation ONLY without save-point? Only found "/jobs/:jobid/savepoints".
  2.  For "/jobs/:jobid/savepoints", how could I form the URL with cancellation and with directory
  3.  Any cancellation progress monitoring in 1.5 like previous version 1.3/1.4?
     *   previous version: /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId

Thank you.


View raw message