flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: 答复: Effort to add SQL / StreamSQL to Flink
Date Mon, 11 Jan 2016 09:12:49 GMT
Thanks for the feedback!

We will start the SQL effort with putting the existing (batch) Table API on
top of Apache Calcite.
>From there we continue to add streaming support for the Table API before we
put a StreamSQL interface on top.

Consolidating the efforts with the CEP library sounds like a good idea to
me.
Maybe it can be nicely integrated with the streaming table API and later as
well with the StreamSQL interface (the StreamSQL dialect is not defined
yet).

@Till: What do you think about adding CEP features to the Table API. From
the CEP design doc, it looks like we need to add a pattern matching
operator in addition to the window features that we need to add for
streaming Table API in any case.

Best, Fabian

2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangsong@huawei.com>:

> I suggest refering to Esper EPL[1], which is a SQL-standard language
> extend to offering a cluster of window, pattern matching.  EPL can both
> support Streaming SQL and CEP with one unified syntax.
>
> [1]
> http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf
>   (Chapter 5. EPL Reference: Clauses)
>
>
> Regards
> Song
>
>
> -----邮件原件-----
> 发件人: Chiwan Park [mailto:chiwanpark@apache.org]
> 发送时间: 2016年1月11日 10:31
> 收件人: dev@flink.apache.org
> 主题: Re: Effort to add SQL / StreamSQL to Flink
>
> We still don’t have a concensus about the streaming SQL and CEP library on
> Flink. Some people want to merge these two libraries. Maybe we have to
> discuss about this in mailing list.
>
> > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
> >
> > What's the relationship between the streaming SQL proposed here and
> > the CEP syntax proposed earlier in the week?
> >
> > On Sunday, January 10, 2016, Henry Saputra <henry.saputra@gmail.com>
> wrote:
> >
> >> Awesome! Thanks for the reply, Fabian.
> >>
> >> - Henry
> >>
> >> On Sunday, January 10, 2016, Fabian Hueske <fhueske@gmail.com
> >> <javascript:;>> wrote:
> >>
> >>> Hi Henry,
> >>>
> >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and a few
> >>> subissues.
> >>> I'll reorganize these and add more issues for the tasks described in
> >>> the design document in the next days.
> >>>
> >>> Thanks, Fabian
> >>>
> >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra <henry.saputra@gmail.com
> >> <javascript:;>
> >>> <javascript:;>>:
> >>>
> >>>> HI Fabian,
> >>>>
> >>>> Have you created JIRA ticket to keep track of this new feature?
> >>>>
> >>>> - Henry
> >>>>
> >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske <fhueske@gmail.com
> >> <javascript:;>
> >>> <javascript:;>> wrote:
> >>>>> Hi everybody,
> >>>>>
> >>>>> in the last days, Timo and I refined the design document for
> >>>>> adding a
> >>>> SQL /
> >>>>> StreamSQL interface on top of Flink that was started by Stephan.
> >>>>>
> >>>>> The document proposes an architecture that is centered around
> >>>>> Apache Calcite. Calcite is an Apache top-level project and
> >>>>> includes a SQL
> >>>> parser,
> >>>>> a semantic validator for relational queries, and a rule- and
> >> cost-based
> >>>>> relational optimizer. Calcite is used by Apache Hive and Apache
> >>>>> Drill (among other projects). In a nutshell, the plan is to
> >>>>> translate Table
> >>> API
> >>>>> and SQL queries into Calcite's relational expression trees,
> >>>>> optimize
> >>>> these
> >>>>> trees, and translate them into DataSet and DataStream programs.The
> >>>> document
> >>>>> breaks down the work into several tasks and subtasks.
> >>>>>
> >>>>> Please review the design document and comment.
> >>>>>
> >>>>> -- >
> >>>>>
> >>>>
> >>>
> >> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP
> >> cp1h2TVqdI/edit?usp=sharing
> >>>>>
> >>>>> Unless there are major concerns with the design, Timo and I want
> >>>>> to
> >>> start
> >>>>> next week to move the current Table API on top of Apache Calcite
> >> (Task
> >>> 1
> >>>> in
> >>>>> the document). The goal of this task is to have the same
> >> functionality
> >>> as
> >>>>> currently, but with Calcite in the translation process. This is
a
> >>>> blocking
> >>>>> task that we hope to complete soon. Afterwards, we can
> >>>>> independently
> >>> work
> >>>>> on different aspects such as extending the Table API, adding a SQL
> >>>>> interface (basically just a parser), integration with external
> >>>>> data sources, better code generation, optimization rules,
> >>>>> streaming
> >> support
> >>>> for
> >>>>> the Table API, StreamSQL, etc..
> >>>>>
> >>>>> Timo and I plan to work on a WIP branch to implement Task 1 and
> >>>>> merge
> >>> it
> >>>> to
> >>>>> the master branch once the task is completed. Of course, everybody
> >>>>> is welcome to contribute to this effort. Please let us know such
> >>>>> that we
> >>> can
> >>>>> coordinate our efforts.
> >>>>>
> >>>>> Thanks,
> >>>>> Fabian
>
> Regards,
> Chiwan Park
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message