Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
Date: Wed, 16 Mar 2016 14:50:42 +0100
Message-ID: 
 <CACDjLVjeyo7TOJNBDgX1pbFwrQCXTx6cRx2ix8a_WCLN3aLqMQ@mail.gmail.com>
Subject: Flink Checkpoint on yarn
From: Simone Robutti <simone.robutti@radicalbit.io>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11c3cd2cc0d393052e2acc1f

--001a11c3cd2cc0d393052e2acc1f
Content-Type: text/plain; charset=UTF-8

Hello,

I'm testing the checkpointing functionality with hdfs as a backend.

For what I can see it uses different checkpointing files and resume the
computation from different points and not from the latest available. This
is to me an unexpected behaviour.

I log every second, for every worker, a counter that is increased by 1 at
each step.

So for example on node-1 the count goes up to 5, then I kill a job manager
or task manager and it resumes from 5 or 4 and it's ok. The next time I
kill a job manager the count is at 15 and it resumes at 14 or 15. Sometimes
it may happen that at a third kill the work resumes at 4 or 5 as if the
checkpoint resumed the second time wasn't there.

Once I even saw it jump forward: the first kill is at 10 and it resumes at
9, the second kill is at 70 and it resumes at 9, the third kill is at 15
but it resumes at 69 as if it resumed from the second kill checkpoint.

This is clearly inconsistent.

Also, in the logs I can find that sometimes it uses a checkpoint file
different from the previous, consistent resume.

What am I doing wrong? Is it a known bug?

--001a11c3cd2cc0d393052e2acc1f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello,<div><br></div><div>I&#39;m testing the checkpointin=
g functionality with hdfs as a backend.</div><div><br></div><div>For what I=
 can see it uses different checkpointing files and resume the computation f=
rom different points and not from the latest available. This is to me an un=
expected behaviour.=C2=A0</div><div><br></div><div>I log every second, for =
every worker, a counter that is increased by 1 at each step.=C2=A0</div><di=
v><br></div><div>So for example on node-1 the count goes up to 5, then I ki=
ll a job manager or task manager and it resumes from 5 or 4 and it&#39;s ok=
. The next time I kill a job manager the count is at 15 and it resumes at 1=
4 or 15. Sometimes it may happen that at a third kill the work resumes at 4=
 or 5 as if the checkpoint resumed the second time wasn&#39;t there.</div><=
div><br></div><div>Once I even saw it jump forward: the first kill is at 10=
 and it resumes at 9, the second kill is at 70 and it resumes at 9, the thi=
rd kill is at 15 but it resumes at 69 as if it resumed from the second kill=
 checkpoint.</div><div><br></div><div>This is clearly inconsistent.</div><d=
iv><br></div><div>Also, in the logs I can find that sometimes it uses a che=
ckpoint file different from the previous, consistent resume.</div><div><br>=
</div><div>What am I doing wrong? Is it a known bug?=C2=A0</div></div>

--001a11c3cd2cc0d393052e2acc1f--