Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAAdrtT2Ar03aVpSJAndEp_kAdaS7YsxavAhju7NsY3RSzNTUqA@mail.gmail.com>
References: 
 <CAAdrtT04o6tU+DHM4632v_8p7QxT7ErFWicpuzFUtfaDMOK9FQ@mail.gmail.com>
	<CALuGr6aidCP8Yu+hc4d=NmXSqWS4rGT1a8nodaJzAk=bUamjeA@mail.gmail.com>
	<CAAdrtT1gTc3TWDiq=AvyWQq2RjmG8Dg9TBG0ut9BfTWFR4+LeA@mail.gmail.com>
	<CALuGr6aMKbkJ+=44bUTg6ex=C1-CVF23CFemtScw7mJ+knw2QA@mail.gmail.com>
	<CANZa=GuGzFuRjhO5Dr4iE4MWgOkTR9qdR5Egejhn3ap75FC2_Q@mail.gmail.com>
	<CE29BF81-62CE-4FE9-A3D1-ABBA8CA8C08D@apache.org>
	<988A74794B4FDA4792872DFDB5789CB06231F155@SZXEMA509-MBX.china.huawei.com>
	<CAAdrtT06do35fKwoXDi9a=SqxVOxuJqD9RRjwcEth1aeNLdE3Q@mail.gmail.com>
	<CAC27z=PX=tLy+q-3t7vdY=g3FjftEQdw-1B7qgkkJPWKR9CTHw@mail.gmail.com>
	<CAAdrtT0bXp7j0PyOEX4orxhr-efq8c7nVMgZmL9WVpSDYuQ=_A@mail.gmail.com>
	<CAAdrtT2Ar03aVpSJAndEp_kAdaS7YsxavAhju7NsY3RSzNTUqA@mail.gmail.com>
Date: Wed, 16 Mar 2016 18:17:09 +0100
Message-ID: 
 <CAJZ2dcXAY8mCHi6BDn_r9C=JRNPXqCRuayU=Ou4mer+JYN=CdA@mail.gmail.com>
Subject: 
 =?UTF-8?B?UmU6IOetlOWkjTogRWZmb3J0IHRvIGFkZCBTUUwgLyBTdHJlYW1TUUwgdG8gRmxpbms=?=
From: Vasiliki Kalavri <vasilikikalavri@gmail.com>
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1141281c15b541052e2daf24

--001a1141281c15b541052e2daf24
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hello everyone,

We are happy to announce that the "tableOnCalcite" branch is finally ready
to be merged.
It essentially provides the existing functionality of the Table API, but
now the translation happens through Apache Calcite.
You can find the changes rebased on top of the current master in [1].
We have removed the prototype streaming Table API functionality, which will
be added back once PR [2] is merged.

We'll go through the changes once more and, if no objections, we would like
to go ahead and merge this.

Cheers,
-Vasia.

[1]: https://github.com/vasia/flink/tree/merge-table
[2]: https://github.com/apache/flink/pull/1770


On 15 January 2016 at 10:59, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi everybody,
>
> as previously announced, I pushed a feature branch called "tableOnCalcite=
"
> to the Flink repository.
> We will use this branch to work on FLINK-3221 and its sub-issues.
>
> Cheers, Fabian
>
> 2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhueske@gmail.com>:
>
> > We haven't defined the StreamSQL syntax yet (and I think it will take
> some
> > time until we are at that point).
> > So we are quite flexible with both featurs.
> >
> > Let's keep this opportunity in mind and coordinate when before making
> > decisions about CEP or StreamSQL.
> >
> > Fabian
> >
> > 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrmann@apache.org>:
> >
> >> First of all, it's a great design document. Looking forward having
> stream
> >> SQL in the foreseeable future :-)
> >>
> >> I think it is a good idea to consolidate stream SQL and CEP in the lon=
g
> >> run. CEP's additional features compared to SQL boil down to pattern
> >> detection. Once we have this, it should be only a question of defining
> the
> >> SQL syntax for event patterns in order to integrate CEP with stream SQ=
L.
> >> Oracle has already defined an extension [1] to detect patterns in a se=
t
> of
> >> table rows. This or Esper's event processing language (EPL) [2] could
> be a
> >> good starting point.
> >>
> >> [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959
> >> [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/
> >>
> >> Cheers,
> >> Till
> >>
> >> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhueske@gmail.com>
> >> wrote:
> >>
> >> > Thanks for the feedback!
> >> >
> >> > We will start the SQL effort with putting the existing (batch) Table
> >> API on
> >> > top of Apache Calcite.
> >> > From there we continue to add streaming support for the Table API
> >> before we
> >> > put a StreamSQL interface on top.
> >> >
> >> > Consolidating the efforts with the CEP library sounds like a good id=
ea
> >> to
> >> > me.
> >> > Maybe it can be nicely integrated with the streaming table API and
> >> later as
> >> > well with the StreamSQL interface (the StreamSQL dialect is not
> defined
> >> > yet).
> >> >
> >> > @Till: What do you think about adding CEP features to the Table API.
> >> From
> >> > the CEP design doc, it looks like we need to add a pattern matching
> >> > operator in addition to the window features that we need to add for
> >> > streaming Table API in any case.
> >> >
> >> > Best, Fabian
> >> >
> >> > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangsong@huawei.com>:
> >> >
> >> > > I suggest refering to Esper EPL[1], which is a SQL-standard langua=
ge
> >> > > extend to offering a cluster of window, pattern matching.  EPL can
> >> both
> >> > > support Streaming SQL and CEP with one unified syntax.
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_re=
ference.pdf
> >> > >   (Chapter 5. EPL Reference: Clauses)
> >> > >
> >> > >
> >> > > Regards
> >> > > Song
> >> > >
> >> > >
> >> > > -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6-----
> >> > > =E5=8F=91=E4=BB=B6=E4=BA=BA: Chiwan Park [mailto:chiwanpark@apache=
.org]
> >> > > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2016=E5=B9=B41=E6=9C=8811=E6=
=97=A5 10:31
> >> > > =E6=94=B6=E4=BB=B6=E4=BA=BA: dev@flink.apache.org
> >> > > =E4=B8=BB=E9=A2=98: Re: Effort to add SQL / StreamSQL to Flink
> >> > >
> >> > > We still don=E2=80=99t have a concensus about the streaming SQL an=
d CEP
> >> library
> >> > on
> >> > > Flink. Some people want to merge these two libraries. Maybe we hav=
e
> to
> >> > > discuss about this in mailing list.
> >> > >
> >> > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimiduk@gmail.com>
> >> wrote:
> >> > > >
> >> > > > What's the relationship between the streaming SQL proposed here
> and
> >> > > > the CEP syntax proposed earlier in the week?
> >> > > >
> >> > > > On Sunday, January 10, 2016, Henry Saputra <
> henry.saputra@gmail.com
> >> >
> >> > > wrote:
> >> > > >
> >> > > >> Awesome! Thanks for the reply, Fabian.
> >> > > >>
> >> > > >> - Henry
> >> > > >>
> >> > > >> On Sunday, January 10, 2016, Fabian Hueske <fhueske@gmail.com
> >> > > >> <javascript:;>> wrote:
> >> > > >>
> >> > > >>> Hi Henry,
> >> > > >>>
> >> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and =
a
> >> few
> >> > > >>> subissues.
> >> > > >>> I'll reorganize these and add more issues for the tasks
> described
> >> in
> >> > > >>> the design document in the next days.
> >> > > >>>
> >> > > >>> Thanks, Fabian
> >> > > >>>
> >> > > >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra <
> henry.saputra@gmail.com
> >> > > >> <javascript:;>
> >> > > >>> <javascript:;>>:
> >> > > >>>
> >> > > >>>> HI Fabian,
> >> > > >>>>
> >> > > >>>> Have you created JIRA ticket to keep track of this new featur=
e?
> >> > > >>>>
> >> > > >>>> - Henry
> >> > > >>>>
> >> > > >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske <
> fhueske@gmail.com
> >> > > >> <javascript:;>
> >> > > >>> <javascript:;>> wrote:
> >> > > >>>>> Hi everybody,
> >> > > >>>>>
> >> > > >>>>> in the last days, Timo and I refined the design document for
> >> > > >>>>> adding a
> >> > > >>>> SQL /
> >> > > >>>>> StreamSQL interface on top of Flink that was started by
> Stephan.
> >> > > >>>>>
> >> > > >>>>> The document proposes an architecture that is centered aroun=
d
> >> > > >>>>> Apache Calcite. Calcite is an Apache top-level project and
> >> > > >>>>> includes a SQL
> >> > > >>>> parser,
> >> > > >>>>> a semantic validator for relational queries, and a rule- and
> >> > > >> cost-based
> >> > > >>>>> relational optimizer. Calcite is used by Apache Hive and
> Apache
> >> > > >>>>> Drill (among other projects). In a nutshell, the plan is to
> >> > > >>>>> translate Table
> >> > > >>> API
> >> > > >>>>> and SQL queries into Calcite's relational expression trees,
> >> > > >>>>> optimize
> >> > > >>>> these
> >> > > >>>>> trees, and translate them into DataSet and DataStream
> >> programs.The
> >> > > >>>> document
> >> > > >>>>> breaks down the work into several tasks and subtasks.
> >> > > >>>>>
> >> > > >>>>> Please review the design document and comment.
> >> > > >>>>>
> >> > > >>>>> -- >
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>
> >> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP
> >> > > >> cp1h2TVqdI/edit?usp=3Dsharing
> >> > > >>>>>
> >> > > >>>>> Unless there are major concerns with the design, Timo and I
> want
> >> > > >>>>> to
> >> > > >>> start
> >> > > >>>>> next week to move the current Table API on top of Apache
> Calcite
> >> > > >> (Task
> >> > > >>> 1
> >> > > >>>> in
> >> > > >>>>> the document). The goal of this task is to have the same
> >> > > >> functionality
> >> > > >>> as
> >> > > >>>>> currently, but with Calcite in the translation process. This
> is
> >> a
> >> > > >>>> blocking
> >> > > >>>>> task that we hope to complete soon. Afterwards, we can
> >> > > >>>>> independently
> >> > > >>> work
> >> > > >>>>> on different aspects such as extending the Table API, adding=
 a
> >> SQL
> >> > > >>>>> interface (basically just a parser), integration with extern=
al
> >> > > >>>>> data sources, better code generation, optimization rules,
> >> > > >>>>> streaming
> >> > > >> support
> >> > > >>>> for
> >> > > >>>>> the Table API, StreamSQL, etc..
> >> > > >>>>>
> >> > > >>>>> Timo and I plan to work on a WIP branch to implement Task 1
> and
> >> > > >>>>> merge
> >> > > >>> it
> >> > > >>>> to
> >> > > >>>>> the master branch once the task is completed. Of course,
> >> everybody
> >> > > >>>>> is welcome to contribute to this effort. Please let us know
> such
> >> > > >>>>> that we
> >> > > >>> can
> >> > > >>>>> coordinate our efforts.
> >> > > >>>>>
> >> > > >>>>> Thanks,
> >> > > >>>>> Fabian
> >> > >
> >> > > Regards,
> >> > > Chiwan Park
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

--001a1141281c15b541052e2daf24--