Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Received-SPF: pass (nike.apache.org: domain of gyula.fora@gmail.com designates
 209.85.215.42 as permitted sender)
MIME-Version: 1.0
References: 
 <CA+faj9wCt4XaFv64xYrkpCdGn6ejhUNDGeuNASxqRi5Cp8uM-w@mail.gmail.com>
 <CAF6bXwUQ3Je8zYFxmdW3N2LfAUS3u+7vs3CeHWBRAJfMScKZpg@mail.gmail.com>
 <CAGco--aQP_mfWf8N-1m5hxScmLDnJ2xjcLgrfMMiVp214iod=Q@mail.gmail.com>
 <CAGr9p8DWfm+LEavueKgqB5i6n3icAr4KVivLtzgQ+Sk7VQXnBg@mail.gmail.com>
 <CANC1h_s8hVVPZBbm5c_C=gnb2X5hr5RwB7f=tEN2D7YgEaq1Bg@mail.gmail.com>
 <CA+faj9xcLuzrsozjR8=F77_UAHOBN5596WtmYAhrCM7i1z4FWQ@mail.gmail.com>
 <A1A79607-7990-4819-B5E9-112D99FA54E8@apache.org>
 <CAGr9p8DW7WHEWNO+JJ9EXqS6BoC_cptZmG=sS3BFZ10u0UZL8w@mail.gmail.com>
In-Reply-To: 
 <CAGr9p8DW7WHEWNO+JJ9EXqS6BoC_cptZmG=sS3BFZ10u0UZL8w@mail.gmail.com>
From: =?UTF-8?Q?Gyula_F=C3=B3ra?= <gyula.fora@gmail.com>
Date: Wed, 01 Jul 2015 09:38:31 +0000
Message-ID: 
 <CA+faj9x+kp_rzmPgNNeefXjpH-vrNDcER-sA+KEjd49EdjOxmg@mail.gmail.com>
Subject: Re: Replacing Checkpointed interface with field annotations
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=089e0160bda490f5f40519cd1669

--089e0160bda490f5f40519cd1669
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I understand your concerns Robert but I don't fully agree.

The Checkpointed interface works indeed but there are so many use cases
that it is not suitable for in the long run, and also the whole interface
is slightly awkward in my opinion when returning simple fields which are
already serializable.

This motivated the introduction of the OperatorStateInterface which you can
call the first rework of the checkpointed interface, but I see that as the
first version which is actually capable of handling many issues that were
obvious with the Checkpointed interfaces.

This is actually not only a rework of the interface but the rework of the
state concept and runtime handling. This needs to be clean if we are moving
streaming out of beta, and should provide the needed funcionality. I think
we can afford to experiment around a little bit with these interfaces and
see the implications for the applications that we can develop with them as
we think of statefulness as a major advantage of Flink streaming.

So actually I think this is the only time when we can afford rework these
interfaces without big costs to make it work for the future.


Robert Metzger <rmetzger@apache.org> ezt =C3=ADrta (id=C5=91pont: 2015. j=
=C3=BAl. 1., Sze,
11:25):

> Whats causing me the biggest headache here is that I don't see an end on
> all these "state interface" reworks.
> I think this is now the third big change to the interface.
> It is a horrible user experience to rework your old code with each new
> Flink release.
>
> I understand that there are always ways to improve interfaces, and I'm su=
re
> Flink has many that we can improve.
> But there are (in my opinion) more important things than reworking the
> interfaces every second week ... for example that the functionality they
> are providing is actually working and well tested.
>
>
>
> On Wed, Jul 1, 2015 at 11:15 AM, Ufuk Celebi <uce@apache.org> wrote:
>
> >
> > On 01 Jul 2015, at 10:57, Gyula F=C3=B3ra <gyula.fora@gmail.com> wrote:
> >
> > > Hey,
> > >
> > > Thanks for the feedback guys:
> > >
> > > @Max: You are right, this is not top priority to changes, I was just
> > > mocking up some alternatives to try to make the state usage even
> simpler
> > so
> > > that the user can keep his current implementations and just add 1-2
> > > annotations.
> >
> > I agree. It's good to cover the "basic" case with a simple solution. :-=
)
> >
> > > @Stephan, Robert: You are right that the checkpointed interface has
> some
> > > advantages from that point of view. Maybe a way to go would be to
> > separate
> > > this signaling functionality (when the checkpoint is taken and maybe
> also
> > > the commits) from the snapshotting itself. One advantage I see there =
is
> > > that we would not need to have 3 different interfaces doing pretty mu=
ch
> > the
> > > same thing (OperatorState - needed for partitioned state and differen=
t
> > > backends/out-of-core, Checkpointed - needed for special actions after
> > > checkpoints, Annotations - checkpointing simple fields natively).
> >
> > I also agree with Stephan and Robert that there are other use cases,
> which
> > require the interfaces. I cannot judge your proposal at this point
> though.
> > I'm eager to hear what the others say who worked on this.
> >
> > =E2=80=93 Ufuk
>

--089e0160bda490f5f40519cd1669--