flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Question about the checkpoint mechanism in Flink.
Date Wed, 02 Nov 2016 09:18:58 GMT
Yes you're right. Whenever you have multiple input channels which could
also be the case if you do a repartitioning between two mappers.

On Tue, Nov 1, 2016 at 11:48 PM, Renjie Liu <liurenjie2008@gmail.com> wrote:

> Hi, Till:
> I think the multiple input should include the more general case where
> redistribution happens between subtasks, right? Since in this case we also
> need to align check barrier.
>
> Till Rohrmann <trohrmann@apache.org>于2016年11月1日周二 下午11:05写道:
>
>> The tuples are not buffered until the snapshot is globally complete (a
>> snapshot is globally complete iff all operators have successfully taken a
>> snapshot). They are only buffered until the corresponding checkpoint
>> barrier on the second input is received. Once this is the case, the
>> checkpoint barrier will directly be send to the downstream operators. Next
>> a snapshot is taken. Depending on the state backend this can happen
>> asynchronously or synchronously. After this is done, the operator continues
>> processing elements (for the first input, the buffered elements are
>> consumed first).
>>
>> With multiple inputs I referred to a coFlatMap operator or a join
>> operator which have both two inputs.
>>
>> Cheers,
>> Till
>>
>> On Tue, Nov 1, 2016 at 3:29 PM, Renjie Liu <liurenjie2008@gmail.com>
>> wrote:
>>
>> Hi, Till:
>> By operator with multiple inputs, do you mean inputs from multiple
>> subtasks?
>>
>> On Tue, Nov 1, 2016 at 8:56 PM Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>> Hi Li,
>>
>> the statement refers to operators with multiple inputs (two in this
>> case). With the current implementation you will indeed block one of the
>> inputs after receiving a checkpoint barrier n until you've received the
>> corresponding checkpoint barrier n on the other input as well. This is what
>> we call checkpoint barrier alignment. If the processing time on both input
>> paths is similar and thus there is no back pressure on any of the inputs,
>> the alignment should not take too long. In case where one of the inputs is
>> considerably slower than the other, you should an additional delay.
>>
>> For single input operators, you don't have to align the checkpoint
>> barriers.
>>
>> The checkpoint barrier alignment is not strictly necessary, but it allows
>> us to not having to store all in flight records from the second input which
>> arrive between the checkpoint barrier on the first input and the
>> corresponding barrier on the second input. We might change this
>> implementation in the future, though.
>>
>> Cheers,
>> Till
>>
>> On Tue, Nov 1, 2016 at 8:05 AM, Li Wang <wangli1426@gmail.com> wrote:
>>
>> Hi all,
>>
>> I have a question regarding to the state checkpoint mechanism in Flink. I
>> find the statement  "Once the last stream has received barrier n, the
>> operator emits all pending outgoing records, and then emits
>> snapshot n barriers itself” on the document https://ci.apache.org/
>> projects/flink/flink-docs-master/internals/stream_
>> checkpointing.html#exactly-once-vs-at-least-once.
>>
>> Does this mean that to achieve exactly-once semantic, instead of sending
>> tuples downstream immediately the operator buffers its outgoing tuples in a
>> pending queue until the current snapshot is committed? If yes, will this
>> introduce significant processing delay?
>>
>> Thanks,
>> Li
>>
>>
>> --
>> Liu, Renjie
>> Software Engineer, MVAD
>>
>>
>> --
> Liu, Renjie
> Software Engineer, MVAD
>

Mime
View raw message