flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vipul singh <neoea...@gmail.com>
Subject Re: Questions about checkpoints/savepoints
Date Tue, 10 Oct 2017 01:01:56 GMT
Thanks Stefan for the answers above. These are really helpful.

I have a few followup questions:

   1. I see my savepoints are created in a folder, which has a _metadata
   file and another file. Looking at the code
   <https://github.com/apache/flink/blob/6642768ad8f8c5d1856742a6d148f7724c20666c/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointStore.java#L191>
   it seems like the metadata file contains tasks states, operator state
   and master states
   <https://github.com/apache/flink/blob/6642768ad8f8c5d1856742a6d148f7724c20666c/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/savepoint/SavepointV2.java#L87>.
   What is the purpose of the other file in the savepoint folder? My guess is
   it should be a checkpoint file?
   2. I am planning to use s3 as my state backend, so want to ensure that
   application restarts are not affected by read-after-write consistency of
   s3( if I use s3 as a savepoint backend). I am curious how flink restores
   data from the _metadata file, and the other file? Does the _metadata file
   contain path to these other files? or would it do a listing on the s3
   folder?


Please let me know,

Thanks,
Vipul

On Tue, Sep 26, 2017 at 2:36 AM, Stefan Richter <s.richter@data-artisans.com
> wrote:

> Hi,
>
> I have answered your questions inline:
>
>
>    1. It seems to me that checkpoints can be treated as flink internal
>    recovery mechanism, and savepoints act more as user-defined recovery
>    points. Would that be a correct assumption?
>
> You could see it that way, but I would describe savepoints more as
> user-defined *restart* points than *recovery* points. Please take a look at
> my answers in this thread, because they cover most of your question:
>
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/difference-between-checkpoints-amp-savepoints-td14787.html .
>
>
>    1. While cancelling an application with -s option, it specifies the
>    savepoint location. Is there a way during application startup to identify
>    the last know savepoint from a folder by itself, and restart from there.
>    Since I am saving my savepoints on s3, I want to avoid issues arising from
>    *ls* command on s3 due to read-after-write consistency of s3.
>
> I don’t think that this feature exists, you have to specify the savepoint.
>
>
>    1. Suppose my application has a checkpoint at point t1, and say i
>    cancel this application sometime in future before the next available
>    checkpoint( say t1+x). If I start the application without specifying the
>    savepoint, it will start from the last known checkpoint(at t1), which wont
>    have the application state saved, since I had cancelled the application.
>    Would this is a correct assumption?
>
> If you restart a canceled application it will not consider checkpoints.
> They are only considered in recovery on failure. You need to specify a
> savepoint or externalized checkpoint for restarts to make explicit that you
> intend to restart a job, and not to run a new instance of the job.
>
>
>    1. Would using ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION be
>    same as manually saving regular savepoints?
>
> Not the same, because checkpoints and savepoints are different in certain
> aspects, but both methods leave you with something that survives job
> cancelation and can be used to restart from a certain state.
>
> Best,
> Stefan
>
>


-- 
Thanks,
Vipul

Mime
View raw message