flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Questions regarding to Flink 1.5.0 REST API change
Date Thu, 21 Jun 2018 07:10:51 GMT
That's quite weird that it tries to us the local file-system. Maybe it 
derives the FS from state.backend.fs.checckpointdir, but uses it for 
savepoints.dir?

What happens if you set state.backend.fs.checkpointdir also to HDFS?


On 21.06.2018 08:07, Siew Wai Yow wrote:
>
> Thanks Chesnay, the application will take value from 
> "state.savepoints.dir" as default if set target-directory to nul. But 
> then it trying to create the directory in local machine, which 
> caused the below error because it is a HDFS directory. The same URL 
> works in previous Flink 1.3.2. Is something break in Flink 1.5.0? Or 
> anything we need to extra configure? Thank you.
>
>
> Caused by: java.io.IOException: *Mkdirs failed to create* 
> hdfs://192.168.56.150:8020/flink/savepoints/savepoint-855552-fb6eca4f3dbb
>         at 
> org.apache.flink.core.fs.local.*LocalFileSystem*.create(LocalFileSystem.java:271)
>         at 
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
>         at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.createStream(FsCheckpointStreamFactory.java:348)
>         ... 25 more
>
>
>
> flink-conf.yaml:
>
> --------------------------------
>
> state.backend: rocksdb
> state.backend.fs.checkpointdir: file:///tmp/rocksdb_simple_example
> state.backend.incremental: true
> state.backend.async: true
> state.checkpoints.num-retained: 5
> state.savepoints.dir: hdfs://192.168.56.150:8020/flink/savepoints
>
>
> -Yow
>
>
>
> ------------------------------------------------------------------------
> *From:* Chesnay Schepler <chesnay@apache.org>
> *Sent:* Wednesday, June 20, 2018 3:15 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
> I think you can set the target-directory to null. But I'm not sure why 
> this particular request requires this, other request allow optional 
> fields to simply be ommitted...
>
> On 20.06.2018 06:12, Siew Wai Yow wrote:
>>
>> Hi all,
>>
>>
>> Seems pass in target-directory is a must now for checkpoints REST 
>> API, and the status will not response with save point directory 
>> anymore. I can pass in but the information is redundant with the same 
>> already defined in flink-config.yaml. May I know is there a way to 
>> retrieve the save point directory from flink-config.yaml in flink 
>> application? I am not able to get it from env.getConfig(). Thank you.
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Chesnay Schepler <chesnay@apache.org> <mailto:chesnay@apache.org>
>> *Sent:* Tuesday, June 19, 2018 11:55 PM
>> *To:* user@flink.apache.org <mailto:user@flink.apache.org>
>> *Subject:* Re: Questions regarding to Flink 1.5.0 REST API change
>> 1. PATCH to /jobs/:jobid, you can specify CANCEL/STOP with the "mode" 
>> query parameter
>>
>> 2. POST to /jobs/:jobid/savepoints, with a json payload. Returns a 
>> trigger id, used for 3).
>> {|
>> "target-directory" : { |
>> |"type" : "string" }, "cancel-job" : { "type" : "boolean" } } |
>> 3. GET to /jobs/:jobid/savepoints/:triggerid
>>
>> On 19.06.2018 17:40, Esteban Serrano wrote:
>>> For #1, you need to use a PATCH request to "*/jobs/:jobid*"
>>>
>>> On Tue, Jun 19, 2018 at 11:35 AM Siew Wai Yow <wai_yow@hotmail.com 
>>> <mailto:wai_yow@hotmail.com>> wrote:
>>>
>>>     Hi,
>>>
>>>
>>>     Regarding to Flink 1.5.0 REST API breaking change,
>>>
>>>       * /The REST API to cancel a job was changed./
>>>       * /The REST API to cancel a job with savepoint was changed./
>>>
>>>     I have few dump questions,
>>>
>>>
>>>      1. Any replacement for cancellation ONLY without save-point?
>>>         Only found "*/jobs/:jobid/savepoints*".
>>>      2. For "*/jobs/:jobid/savepoints*", how could I form the URL
>>>         with cancellation and with directory afterward?
>>>          1. http://192.168.56.151:8081/jobs/<jobid>/savepoints/cancel-job/true??
>>>
>>>      3. Any cancellation progress monitoring in 1.5 like previous
>>>         version 1.3/1.4?
>>>          1. previous version:
>>>             /jobs/:jobid/cancel-with-savepoint/in-progress/:requestId
>>>
>>>
>>>     Thank you.
>>>
>>>     Regards,
>>>     Yow
>>>
>>>
>>
>


Mime
View raw message