flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Why checkpointing is synchronized with data processing?
Date Mon, 31 Oct 2016 14:03:24 GMT
Hi!

Checkpoints do not really hold data processing. The only synchronous call
is to create a "state snapshot". Some state backends implement that
synchronously (like the FsStateBackend) some asynchronously
(RocksDBStateBackend).

It depends on the state backend how much is synchronous (blocking) vs
asynchronous. The checkpoint algorithm itself allows for asynchronous
persistence.

Stephan


On Sat, Oct 29, 2016 at 6:05 AM, Renjie Liu <liurenjie2008@gmail.com> wrote:

> Hi, flink dev team:
>
> I'm reading flink's source code and it seems that the process of saving
> checkpointing data to external storage system will block data processing,
> and I wonder why this has to be designed like this? Isn't is a potential
> risk that may increase latency of processing?
>
> Is the following strategy better?
>
>    1. Save state data to local disk
>    2. Resume data processing
>    3. Saving state data to external system asynchronously
>
> --
> Liu, Renjie
> Software Engineer, MVAD
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message