Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAKiyyaEA3dm71bbe6VrtjFhsOONJwRN=yywKb05tg67cV7e9ig@mail.gmail.com>
References: <CAFDmHdNnZkKNy4zXXhYn34UkA4pBPtW1thqKWD4sRB0Qs5aLqw@mail.gmail.com>
 <CAFDmHdMZtarS2GeZcrF+EnBROzp5jt1Brhcb6y=fRC85VB4BUA@mail.gmail.com> <CAKiyyaEA3dm71bbe6VrtjFhsOONJwRN=yywKb05tg67cV7e9ig@mail.gmail.com>
From: Eron Wright <eronwright@gmail.com>
Date: Thu, 14 Sep 2017 11:03:12 -0700
Message-ID: <CAGKOEz2pToXJ4TKHOE8_8JPq-2VbBDoFMUY-W1p0D4e=qqfwxg@mail.gmail.com>
Subject: Re: Flink flick cancel vs stop
To: Ufuk Celebi <uce@apache.org>
Cc: Elias Levy <fearsome.lucidity@gmail.com>, user <user@flink.apache.org>,
	Till Rohrmann <trohrmann@apache.org>
Content-Type: multipart/alternative; boundary="001a113d840006df3d05592a1750"
archived-at: Thu, 14 Sep 2017 18:03:25 -0000

--001a113d840006df3d05592a1750
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I too am curious about stop vs cancel.  I'm trying to understand the
motivations a bit more.

The current behavior of stop is basically that the sources become bounded,
leading to the job winding down.

The interesting question is how best to support 'planned' maintenance
procedures such as app upgrade and scale changes.   I think a good
enhancement could be to stop precisely at checkpoint time to prevent
emission of spurious records.  Today the behavior of 'cancel w/ savepoint'
is at-least-once because the two operations aren't atomic.  Earlier I had
assumed that 'stop' would evolve in this direction but I suppose we could
improve the atomicity of 'cancel /w savepoint' rather than implicating
'stop'.

A different direction for 'stop' might be to improve the determinism of
bounding a streaming job such that the stop point is well-understood in
terms of the source.  For example, stopping at a offset provided as a stop
parameter.   Today I suppose one would rely on external state to remember
the stop point, e.g. FlinkKafkaConsumer010::setStartFromGroupOffsets.

On Thu, Sep 14, 2017 at 1:03 AM, Ufuk Celebi <uce@apache.org> wrote:

> Hey Elias,
>
> sorry for the delay here. No, stop is not deprecated but not fully
> implemented yet. One missing part is migration of the existing source
> functions as you say.
>
> Let me pull in Till for more details on this. @Till: Is there more
> missing than migrating the sources?
>
> Here is the PR and discussion for reference:
> https://github.com/apache/flink/pull/750
>
> I would also really love to see this fully implemented in Flink. I
> don't expect this to happen for the upcoming 1.4 release though.
>
> =E2=80=93 Ufuk
>
>
> On Wed, Sep 13, 2017 at 7:07 PM, Elias Levy <fearsome.lucidity@gmail.com>
> wrote:
> > Anyone?
> >
> > On Mon, Sep 11, 2017 at 6:17 PM, Elias Levy <fearsome.lucidity@gmail.co=
m
> >
> > wrote:
> >>
> >> I was wondering about the status of the flink stop command.  At first
> >> blush it would seem as the preferable way to shutdown a Flink job, but
> it
> >> depends on StoppableFunction being implemented by sources and I notice
> that
> >> the Kafka source does not seem to implement it.  In addition, the
> command
> >> does not -s  --withSavepoint like cancel does.
> >>
> >> Is stop deprecated?
> >
> >
>

--001a113d840006df3d05592a1750
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I too am curious about stop vs cancel.=C2=A0 I&#39;m =
trying to understand the motivations a bit more.</div><div><br><div>The cur=
rent behavior of stop is basically that the sources become bounded, leading=
 to the job winding down.</div><div><br></div><div>The interesting question=
 is how best to support &#39;planned&#39; maintenance procedures such as ap=
p upgrade and scale changes. =C2=A0 I think a good enhancement could be to =
stop precisely at checkpoint time to prevent emission of spurious records.=
=C2=A0 Today the behavior of &#39;cancel w/ savepoint&#39; is at-least-once=
 because the two operations aren&#39;t atomic.=C2=A0 Earlier I had assumed =
that &#39;stop&#39; would evolve in this direction but I suppose we could i=
mprove the atomicity of &#39;cancel /w savepoint&#39; rather than implicati=
ng &#39;stop&#39;.</div><div><br></div><div>A different direction for &#39;=
stop&#39; might be to improve the determinism of bounding a streaming job s=
uch that the stop point is well-understood in terms of the source.=C2=A0 Fo=
r example, stopping at a offset provided as a stop parameter. =C2=A0 Today =
I suppose one would rely on external state to remember the stop point, e.g.=
 FlinkKafkaConsumer010::setStartFromGroupOffsets.</div></div></div><div cla=
ss=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Sep 14, 2017 at 1=
:03 AM, Ufuk Celebi <span dir=3D"ltr">&lt;<a href=3D"mailto:uce@apache.org"=
 target=3D"_blank">uce@apache.org</a>&gt;</span> wrote:<br><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex">Hey Elias,<br>
<br>
sorry for the delay here. No, stop is not deprecated but not fully<br>
implemented yet. One missing part is migration of the existing source<br>
functions as you say.<br>
<br>
Let me pull in Till for more details on this. @Till: Is there more<br>
missing than migrating the sources?<br>
<br>
Here is the PR and discussion for reference:<br>
<a href=3D"https://github.com/apache/flink/pull/750" rel=3D"noreferrer" tar=
get=3D"_blank">https://github.com/apache/<wbr>flink/pull/750</a><br>
<br>
I would also really love to see this fully implemented in Flink. I<br>
don&#39;t expect this to happen for the upcoming 1.4 release though.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
=E2=80=93 Ufuk<br>
</font></span><div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
On Wed, Sep 13, 2017 at 7:07 PM, Elias Levy &lt;<a href=3D"mailto:fearsome.=
lucidity@gmail.com">fearsome.lucidity@gmail.com</a>&gt; wrote:<br>
&gt; Anyone?<br>
&gt;<br>
&gt; On Mon, Sep 11, 2017 at 6:17 PM, Elias Levy &lt;<a href=3D"mailto:fear=
some.lucidity@gmail.com">fearsome.lucidity@gmail.com</a>&gt;<br>
&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt; I was wondering about the status of the flink stop command.=C2=A0 =
At first<br>
&gt;&gt; blush it would seem as the preferable way to shutdown a Flink job,=
 but it<br>
&gt;&gt; depends on StoppableFunction being implemented by sources and I no=
tice that<br>
&gt;&gt; the Kafka source does not seem to implement it.=C2=A0 In addition,=
 the command<br>
&gt;&gt; does not -s=C2=A0 --withSavepoint like cancel does.<br>
&gt;&gt;<br>
&gt;&gt; Is stop deprecated?<br>
&gt;<br>
&gt;<br>
</div></div></blockquote></div><br></div>

--001a113d840006df3d05592a1750--