flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Wang <wangli1...@gmail.com>
Subject Re: Question about the checkpoint mechanism in Flink.
Date Tue, 01 Nov 2016 14:03:48 GMT

Hi Till,

Thanks for your prompt reply. I understand that input streams should be aligned such that
a consistent state snapshot can be generated. In my opinion, that statement indicates that
an operator will buffer its output tuples until the snapshot is committed. I am wondering
if my understand to that very statement is right. If yes, why an operator should pend its
output tuples? Is that for replaying output tuples during the state recovery of an downstream
operator?

Thanks and regards,
Li


Sent from my iPhone

> On 1 Nov 2016, at 8:56 PM, Till Rohrmann <trohrmann@apache.org> wrote:
> 
> Hi Li,
> 
> the statement refers to operators with multiple inputs (two in this case). With the current
implementation you will indeed block one of the inputs after receiving a checkpoint barrier
n until you've received the corresponding checkpoint barrier n on the other input as well.
This is what we call checkpoint barrier alignment. If the processing time on both input paths
is similar and thus there is no back pressure on any of the inputs, the alignment should not
take too long. In case where one of the inputs is considerably slower than the other, you
should an additional delay.
> 
> For single input operators, you don't have to align the checkpoint barriers.
> 
> The checkpoint barrier alignment is not strictly necessary, but it allows us to not having
to store all in flight records from the second input which arrive between the checkpoint barrier
on the first input and the corresponding barrier on the second input. We might change this
implementation in the future, though.
> 
> Cheers,
> Till
> 
>> On Tue, Nov 1, 2016 at 8:05 AM, Li Wang <wangli1426@gmail.com> wrote:
>> Hi all,
>> 
>> I have a question regarding to the state checkpoint mechanism in Flink. I find the
statement  "Once the last stream has received barrier n, the operator emits all pending outgoing
records, and then emits snapshot n barriers itself” on the document https://ci.apache.org/projects/flink/flink-docs-master/internals/stream_checkpointing.html#exactly-once-vs-at-least-once.
>> 
>> Does this mean that to achieve exactly-once semantic, instead of sending tuples downstream
immediately the operator buffers its outgoing tuples in a pending queue until the current
snapshot is committed? If yes, will this introduce significant processing delay?
>> 
>> Thanks,
>> Li
>> 
> 

Mime
View raw message