Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Received-SPF: pass (athena.apache.org: domain of ewenstephan@gmail.com
 designates 209.85.213.176 as permitted sender)
MIME-Version: 1.0
Sender: ewenstephan@gmail.com
In-Reply-To: 
 <CAGWx-_ufP5wajMhMNst4AMkxR==EE=r3z4=aTNAE12gB8_kFww@mail.gmail.com>
References: <FB13BE46-A8F1-4B0A-B3B6-49E242ED0D07@apache.org>
	<CAGWx-_ufP5wajMhMNst4AMkxR==EE=r3z4=aTNAE12gB8_kFww@mail.gmail.com>
Date: Tue, 31 Mar 2015 18:16:37 +0200
Message-ID: 
 <CANC1h_s+AZ4YhYtMuwpEcxDWfHN+AosfP=+nhYATvHreedQs_w@mail.gmail.com>
Subject: Re: [DISCUSS] Inconsistent naming of intermediate results
From: Stephan Ewen <sewen@apache.org>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=047d7bd75b344e0c8a051297ec59

--047d7bd75b344e0c8a051297ec59
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I like getting the consistency in there.

I was never thinking of the intermediate data sets to be strictly produced
by a vertex, so I am unsure whether we should use that exact naming scheme,
or one that disconnects the results from the term "VertexResult".

On Tue, Mar 31, 2015 at 5:27 PM, Kostas Tzoumas <ktzoumas@apache.org> wrote=
:

> I like the fact that the naming scheme follows some logic.
>
> I also like that we have two easy to understand concepts:
> - Operator (be that in any of the above representations)
> - Result (of executing an operator)
>
> +1
>
> On Tue, Mar 31, 2015 at 4:50 PM, Ufuk Celebi <uce@apache.org> wrote:
>
> > On a high level we call intermediate data produced by programs
> > "intermediate results". For example in a WordCount map-reduce program t=
he
> > map function produces an intermediate result, which consists of (word, =
1)
> > pairs and the reduce function consumes this intermediate result. Kostas
> has
> > recently added documentation explaining the core concepts [1].
> >
> > The naming of classes related to intermediate results is inconsistent
> (and
> > probably confusing).
> >
> > - In JobGraphs (internal low-level API to define programs) they are
> called
> > IntermediateDataSet and identified by IntermediateDataSetIDs.
> >
> > - In ExecutionGraphs (JobManager structure used for state
> > tracking/scheduling) they are called IntermediateResult at the
> > ExecutionJobVertex (identified by IntermediateDataSetID) and
> > IntermediateResultPartition at the ExecutionVertex (identified by
> > IntermediateResultPartitionID).
> >
> > - At runtime (TaskManager) they are called ResultPartition and identifi=
ed
> > by ResultPartitionID (composition of ExecutionAttemptID and
> > IntermediateResultPartitionID). These are further subpartitioned into
> > ResultSubpartition instances.
> >
> > I propose to get the naming more in line with the existing naming schem=
e
> > and prefix it with the corresponding managemenet structures:
> >
> > 1) IntermediateDataSet =3D> JobVertexResult (identified by
> JobVertexResultID)
> > 2) IntermediateResult =3D> ExecutionJobVertexResult (identified by
> > JobVertexResultID)
> > 3) IntermediateResultPartition =3D> ExecutionVertexResult (identified b=
y
> > ExecutionVertexResultID)
> > 4) ResultPartition =3D> Result
> > 5) ResultSubpartition =3D> ResultPartition
> >
> > These names are non-user facing, but still at the core of the system. I
> > think that consistent naming of these classes will make it easier for n=
ew
> > contributors to get an overview of how single components relate to each
> > other (the prefixes indicate this). In the docs, we can still refer to
> the
> > high-level concept as "intermediate results".
> >
> > What's your opinion on this? I think now is a good time to think about
> > this stuff, because the core classes have only been added recently to t=
he
> > system. Feel free to propose alternatives. :-)
> >
> > =E2=80=93 Ufuk
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+t=
asks
>

--047d7bd75b344e0c8a051297ec59--