Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Received-SPF: pass (nike.apache.org: domain of balassi.marton@gmail.com
 designates 209.85.212.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+faj9xNLRKFV0kF1p_K+Qd5=-THwmhL5OTJnxiaVzFG8+aaSQ@mail.gmail.com>
References: 
 <CA+faj9xTyyNw=sP+kK=iY-+RYKUhdKi2XQThzPoNojNc7b=jqQ@mail.gmail.com>
 <5017E5D7-42DC-401F-B89E-4FA48764F002@apache.org>
 <CA+faj9xNLRKFV0kF1p_K+Qd5=-THwmhL5OTJnxiaVzFG8+aaSQ@mail.gmail.com>
From: =?UTF-8?Q?M=C3=A1rton_Balassi?= <balassi.marton@gmail.com>
Date: Wed, 10 Jun 2015 10:08:09 +0200
Message-ID: 
 <CAKADb_MOiX9UFaWibVr5kzvH7H0G_UoEZ=iKvGU1eDM261VO2w@mail.gmail.com>
Subject: Re: Force enabling checkpoints for iterative streaming jobs
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11c385685e58e805182561f6

--001a11c385685e58e805182561f6
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I agree that for the sake of the above mentioned use cases it is reasonable
to add this to the release with the right documentation, for machine
learning potentially loosing one round of feedback data should not matter.

Let us not block prominent users until the next release on this.

On Wed, Jun 10, 2015 at 8:09 AM, Gyula F=C3=B3ra <gyula.fora@gmail.com> wro=
te:

> As for people currently suffering from it:
>
> An application King is developing requires iterations, and they need
> checkpoints. Practically all SAMOA programs would need this.
>
> It is very likely that the state interfaces will be changed after the
> release, so this is not something that we can just add later. I don't see=
 a
> reason why we should not add it, as it is clearly documented. In this
> actual case not having guarantees at all means people will never use it i=
n
> any production system. Having limited guarantees means that it will depen=
d
> on the application.
>
> On Wed, Jun 10, 2015 at 12:53 AM, Ufuk Celebi <uce@apache.org> wrote:
>
> > Hey Gyula,
> >
> > I understand your reasoning, but I don't think its worth to rush this
> into
> > the release.
> >
> > As you've said, we cannot give precise guarantees. But this is arguably
> > one of the key requirements for any fault tolerance mechanism. Therefor=
e
> I
> > disagree that this is better than not having anything at all. I think i=
t
> > will already go a long way to have the non-iterative case working
> reliably.
> >
> > And as far as I know there are no users really suffering from this at t=
he
> > moment (in the sense that someone has complained on the mailing list).
> >
> > Hence, I vote to postpone this.
> >
> > =E2=80=93 Ufuk
> >
> > On 10 Jun 2015, at 00:19, Gyula F=C3=B3ra <gyfora@apache.org> wrote:
> >
> > > Hey all,
> > >
> > > It is currently impossible to enable state checkpointing for iterativ=
e
> > > jobs, because en exception is thrown when creating the jobgraph. This
> > > behaviour is motivated by the lack of precise guarantees that we can
> give
> > > with the current fault-tolerance implementations for cyclic graphs.
> > >
> > > This PR <https://github.com/apache/flink/pull/812> adds an optional
> > flag to
> > > force checkpoints even in case of iterations. The algorithm will take
> > > checkpoints periodically as before, but records in transit inside the
> > loop
> > > will be lost.
> > >
> > > However even this guarantee is enough for most applications (Machine
> > > Learning for instance) and certainly much better than not having
> anything
> > > at all.
> > >
> > >
> > > I suggest we add this to the 0.9 release as currently many applicatio=
ns
> > > suffer from this limitation (SAMOA, ML pipelines, graph streaming etc=
.)
> > >
> > >
> > > Cheers,
> > >
> > > Gyula
> >
> >
>

--001a11c385685e58e805182561f6--