flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: 答复: Effort to add SQL / StreamSQL to Flink
Date Fri, 15 Jan 2016 09:59:58 GMT
Hi everybody,

as previously announced, I pushed a feature branch called "tableOnCalcite"
to the Flink repository.
We will use this branch to work on FLINK-3221 and its sub-issues.

Cheers, Fabian

2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhueske@gmail.com>:

> We haven't defined the StreamSQL syntax yet (and I think it will take some
> time until we are at that point).
> So we are quite flexible with both featurs.
>
> Let's keep this opportunity in mind and coordinate when before making
> decisions about CEP or StreamSQL.
>
> Fabian
>
> 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrmann@apache.org>:
>
>> First of all, it's a great design document. Looking forward having stream
>> SQL in the foreseeable future :-)
>>
>> I think it is a good idea to consolidate stream SQL and CEP in the long
>> run. CEP's additional features compared to SQL boil down to pattern
>> detection. Once we have this, it should be only a question of defining the
>> SQL syntax for event patterns in order to integrate CEP with stream SQL.
>> Oracle has already defined an extension [1] to detect patterns in a set of
>> table rows. This or Esper's event processing language (EPL) [2] could be a
>> good starting point.
>>
>> [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959
>> [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/
>>
>> Cheers,
>> Till
>>
>> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhueske@gmail.com>
>> wrote:
>>
>> > Thanks for the feedback!
>> >
>> > We will start the SQL effort with putting the existing (batch) Table
>> API on
>> > top of Apache Calcite.
>> > From there we continue to add streaming support for the Table API
>> before we
>> > put a StreamSQL interface on top.
>> >
>> > Consolidating the efforts with the CEP library sounds like a good idea
>> to
>> > me.
>> > Maybe it can be nicely integrated with the streaming table API and
>> later as
>> > well with the StreamSQL interface (the StreamSQL dialect is not defined
>> > yet).
>> >
>> > @Till: What do you think about adding CEP features to the Table API.
>> From
>> > the CEP design doc, it looks like we need to add a pattern matching
>> > operator in addition to the window features that we need to add for
>> > streaming Table API in any case.
>> >
>> > Best, Fabian
>> >
>> > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangsong@huawei.com>:
>> >
>> > > I suggest refering to Esper EPL[1], which is a SQL-standard language
>> > > extend to offering a cluster of window, pattern matching.  EPL can
>> both
>> > > support Streaming SQL and CEP with one unified syntax.
>> > >
>> > > [1]
>> > >
>> >
>> http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf
>> > >   (Chapter 5. EPL Reference: Clauses)
>> > >
>> > >
>> > > Regards
>> > > Song
>> > >
>> > >
>> > > -----邮件原件-----
>> > > 发件人: Chiwan Park [mailto:chiwanpark@apache.org]
>> > > 发送时间: 2016年1月11日 10:31
>> > > 收件人: dev@flink.apache.org
>> > > 主题: Re: Effort to add SQL / StreamSQL to Flink
>> > >
>> > > We still don’t have a concensus about the streaming SQL and CEP
>> library
>> > on
>> > > Flink. Some people want to merge these two libraries. Maybe we have to
>> > > discuss about this in mailing list.
>> > >
>> > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimiduk@gmail.com>
>> wrote:
>> > > >
>> > > > What's the relationship between the streaming SQL proposed here and
>> > > > the CEP syntax proposed earlier in the week?
>> > > >
>> > > > On Sunday, January 10, 2016, Henry Saputra <henry.saputra@gmail.com
>> >
>> > > wrote:
>> > > >
>> > > >> Awesome! Thanks for the reply, Fabian.
>> > > >>
>> > > >> - Henry
>> > > >>
>> > > >> On Sunday, January 10, 2016, Fabian Hueske <fhueske@gmail.com
>> > > >> <javascript:;>> wrote:
>> > > >>
>> > > >>> Hi Henry,
>> > > >>>
>> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099
and a
>> few
>> > > >>> subissues.
>> > > >>> I'll reorganize these and add more issues for the tasks described
>> in
>> > > >>> the design document in the next days.
>> > > >>>
>> > > >>> Thanks, Fabian
>> > > >>>
>> > > >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra <henry.saputra@gmail.com
>> > > >> <javascript:;>
>> > > >>> <javascript:;>>:
>> > > >>>
>> > > >>>> HI Fabian,
>> > > >>>>
>> > > >>>> Have you created JIRA ticket to keep track of this new
feature?
>> > > >>>>
>> > > >>>> - Henry
>> > > >>>>
>> > > >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske <fhueske@gmail.com
>> > > >> <javascript:;>
>> > > >>> <javascript:;>> wrote:
>> > > >>>>> Hi everybody,
>> > > >>>>>
>> > > >>>>> in the last days, Timo and I refined the design document
for
>> > > >>>>> adding a
>> > > >>>> SQL /
>> > > >>>>> StreamSQL interface on top of Flink that was started
by Stephan.
>> > > >>>>>
>> > > >>>>> The document proposes an architecture that is centered
around
>> > > >>>>> Apache Calcite. Calcite is an Apache top-level project
and
>> > > >>>>> includes a SQL
>> > > >>>> parser,
>> > > >>>>> a semantic validator for relational queries, and a
rule- and
>> > > >> cost-based
>> > > >>>>> relational optimizer. Calcite is used by Apache Hive
and Apache
>> > > >>>>> Drill (among other projects). In a nutshell, the plan
is to
>> > > >>>>> translate Table
>> > > >>> API
>> > > >>>>> and SQL queries into Calcite's relational expression
trees,
>> > > >>>>> optimize
>> > > >>>> these
>> > > >>>>> trees, and translate them into DataSet and DataStream
>> programs.The
>> > > >>>> document
>> > > >>>>> breaks down the work into several tasks and subtasks.
>> > > >>>>>
>> > > >>>>> Please review the design document and comment.
>> > > >>>>>
>> > > >>>>> -- >
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP
>> > > >> cp1h2TVqdI/edit?usp=sharing
>> > > >>>>>
>> > > >>>>> Unless there are major concerns with the design, Timo
and I want
>> > > >>>>> to
>> > > >>> start
>> > > >>>>> next week to move the current Table API on top of
Apache Calcite
>> > > >> (Task
>> > > >>> 1
>> > > >>>> in
>> > > >>>>> the document). The goal of this task is to have the
same
>> > > >> functionality
>> > > >>> as
>> > > >>>>> currently, but with Calcite in the translation process.
This is
>> a
>> > > >>>> blocking
>> > > >>>>> task that we hope to complete soon. Afterwards, we
can
>> > > >>>>> independently
>> > > >>> work
>> > > >>>>> on different aspects such as extending the Table API,
adding a
>> SQL
>> > > >>>>> interface (basically just a parser), integration with
external
>> > > >>>>> data sources, better code generation, optimization
rules,
>> > > >>>>> streaming
>> > > >> support
>> > > >>>> for
>> > > >>>>> the Table API, StreamSQL, etc..
>> > > >>>>>
>> > > >>>>> Timo and I plan to work on a WIP branch to implement
Task 1 and
>> > > >>>>> merge
>> > > >>> it
>> > > >>>> to
>> > > >>>>> the master branch once the task is completed. Of course,
>> everybody
>> > > >>>>> is welcome to contribute to this effort. Please let
us know such
>> > > >>>>> that we
>> > > >>> can
>> > > >>>>> coordinate our efforts.
>> > > >>>>>
>> > > >>>>> Thanks,
>> > > >>>>> Fabian
>> > >
>> > > Regards,
>> > > Chiwan Park
>> > >
>> > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message