Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
References: 
 <CAOJWaGWyFo-vW79WJVBWaRCypAzGb=Lc9as6vPcUBzBR_o7DbQ@mail.gmail.com>
 <CANC1h_tB0pRB_NTtThcumANuHVjRXmOtq+E6B-bAQ5qHodCViw@mail.gmail.com>
In-Reply-To: 
 <CANC1h_tB0pRB_NTtThcumANuHVjRXmOtq+E6B-bAQ5qHodCViw@mail.gmail.com>
From: Don Frascuchon <frascuchon@gmail.com>
Date: Wed, 13 Jan 2016 11:25:11 +0000
Message-ID: 
 <CAOJWaGWumgtM2pOUsgozcPat4nysyhmcX8gmYQ81cpa2sB1niA@mail.gmail.com>
Subject: Re: Checkpoint for exact-once proccessing
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11444b90f680ae0529356c72

--001a11444b90f680ae0529356c72
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Stephan,

Thanks for your quickly response.

So, consider an operator task with two processed records and no barrier
incoming. If the task fail and must be records, the last consistent
snapshot will be used, which no includes information about the processed
but no checkpointed  records. What about this situation? The registers will
be resent to failed task after, or will be discarded? How flink manage
information about this records for exact-once guarantees? The user function
inside operator must be idempotent (i think about some kind of persistence
in  a sink task)

Thanks in advance !


El mi=C3=A9., 13 ene. 2016 a las 11:17, Stephan Ewen (<sewen@apache.org>)
escribi=C3=B3:

> Hi!
>
> I think there is a misunderstanding. There are no identifiers maintained
> and no individual records deleted.
>
> On recovery, all operators reset their state to a consistent snapshot:
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/st=
ream_checkpointing.html
>
>
> Greetings,
> Stephan
>
>
> On Wed, Jan 13, 2016 at 11:08 AM, Don Frascuchon <frascuchon@gmail.com>
> wrote:
>
>> Hello,
>>
>> I'm trying to understand the process of checkpoint processing for
>> exact-once in Flink, and I have some doubts.
>>
>> The documentation says that when there is a failure and the state of an
>> operator is restored, the already processed records are deleted based on
>> their identifiers.
>>
>> My doubts is, how these identifiers between two checkpoints are
>> maintained? Every time a new input record comes to the stateful operator=
,
>> Flink persists it before making the checkpoint? Otherwise, there may be
>> messages to reprocess after a failure.
>>
>> Thanks !!!
>>
>
>

--001a11444b90f680ae0529356c72
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Stephan,</div><div><br></div>Thanks for your quick=
ly response.<div><br></div><div>So, consider an operator task with two proc=
essed records and no barrier incoming. If the task fail and must be records=
, the last consistent snapshot will be used, which no includes information =
about the processed but no checkpointed =C2=A0records. What about this situ=
ation? The registers will be resent to failed task after, or will be discar=
ded? How flink manage information about this records for exact-once=C2=A0<s=
pan style=3D"color:rgb(51,51,51);font-size:14px;line-height:20px">guarantee=
s? The user function inside operator must be idempotent (i think about some=
 kind of persistence in=C2=A0</span>=C2=A0a sink task)</div><div><br></div>=
<div>Thanks in advance !</div><div><br></div></div><br><div class=3D"gmail_=
quote"><div dir=3D"ltr">El mi=C3=A9., 13 ene. 2016 a las 11:17, Stephan Ewe=
n (&lt;<a href=3D"mailto:sewen@apache.org">sewen@apache.org</a>&gt;) escrib=
i=C3=B3:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi!<br><b=
r>I think there is a misunderstanding. There are no identifiers maintained =
and no individual records deleted.<div><br></div><div>On recovery, all oper=
ators reset their state to a consistent snapshot:=C2=A0<a href=3D"https://c=
i.apache.org/projects/flink/flink-docs-release-0.10/internals/stream_checkp=
ointing.html" target=3D"_blank">https://ci.apache.org/projects/flink/flink-=
docs-release-0.10/internals/stream_checkpointing.html</a></div><div><br></d=
iv><div><br></div><div>Greetings,</div><div>Stephan</div><div><br></div></d=
iv><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Jan 13=
, 2016 at 11:08 AM, Don Frascuchon <span dir=3D"ltr">&lt;<a href=3D"mailto:=
frascuchon@gmail.com" target=3D"_blank">frascuchon@gmail.com</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello,=C2=A0<div=
><br></div><div><div>I&#39;m trying to understand the process of checkpoint=
 processing for exact-once in Flink, and I have some doubts.</div><div><br>=
</div><div>The documentation says that when there is a failure and the stat=
e of an operator is restored, the already processed records are deleted bas=
ed on their identifiers.</div><div><br></div><div>My doubts is, how these i=
dentifiers between two checkpoints are maintained? Every time a new input r=
ecord comes to the stateful operator, Flink persists it before making the c=
heckpoint? Otherwise, there may be messages to reprocess after a failure.</=
div></div><div><br></div><div>Thanks !!!</div></div>
</blockquote></div><br></div>
</blockquote></div>

--001a11444b90f680ae0529356c72--