flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Why checkpointing is synchronized with data processing?
Date Mon, 31 Oct 2016 14:03:24 GMT

Checkpoints do not really hold data processing. The only synchronous call
is to create a "state snapshot". Some state backends implement that
synchronously (like the FsStateBackend) some asynchronously

It depends on the state backend how much is synchronous (blocking) vs
asynchronous. The checkpoint algorithm itself allows for asynchronous


On Sat, Oct 29, 2016 at 6:05 AM, Renjie Liu <liurenjie2008@gmail.com> wrote:

> Hi, flink dev team:
> I'm reading flink's source code and it seems that the process of saving
> checkpointing data to external storage system will block data processing,
> and I wonder why this has to be designed like this? Isn't is a potential
> risk that may increase latency of processing?
> Is the following strategy better?
>    1. Save state data to local disk
>    2. Resume data processing
>    3. Saving state data to external system asynchronously
> --
> Liu, Renjie
> Software Engineer, MVAD

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message