flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jincheng sun <sunjincheng...@gmail.com>
Subject Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP
Date Tue, 13 Jun 2017 07:45:34 GMT
Hi Jark, Dian,

Thanks for bring up this discuss and share the prototype.

+1 to push this great feature forward!

Cheers,
SunJincheng

2017-06-13 15:34 GMT+08:00 Jark Wu <jark@apache.org>:

> Thank you Yueting for pointing out the mistake in the prototype. I
> accidentally introduced it when merge code.
>
> I'm so glad to see so many people are interested in the feature. Let's work
> out together to push it forward!
>
> Cheers,
> Jark Wu
>
>
> 2017-06-13 15:27 GMT+08:00 Liangfei Su <suliangfei@gmail.com>:
>
> > +1 for the feature. Myself was a user of Siddhi, this is pretty user
> > friendly feature to provide to user.
> >
> > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <dian0511.fu@gmail.com> wrote:
> >
> > > Hi Yueting & Dawid Wysakowicz,
> > >
> > > Very glad that you're interested in this feature and you're definitely
> > > welcome to join this work and also anyone else.:)
> > >
> > > Best regards,
> > > Dian
> > >
> > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > wysakowicz.dawid@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Integrating SQL with CEP seems like a really nice idea.
> Unfortunately I
> > > had
> > > > time just for a brief look at the design doc, but it looks really
> cool
> > > and
> > > > thorough. Also will have a second run tomorrow and will try to
> provide
> > > more
> > > > comments. Anyway will be glad to help pushing the initiative forward.
> > > >
> > > > Z pozdrowieniami! / Cheers!
> > > >
> > > > Dawid Wysakowicz
> > > >
> > > > *Data/Software Engineer*
> > > >
> > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > >
> > > > <http://getindata.com/>
> > > >
> > > > 2017-06-13 8:19 GMT+02:00 yueting chen <yestinmail@gmail.com>:
> > > >
> > > > > Hi Dian & Jark,
> > > > >
> > > > > I checked out your prototype code, but it didn't pass the CEPITCase
> > > test
> > > > in
> > > > > the flink-table component.
> > > > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> > > should
> > > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > > instead
> > > > > of
> > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > >
> > > > > I've also read your desgin document and it looks fine to me.
> > Actually,
> > > I
> > > > am
> > > > > working on the same thing recently, I think maybe we can work
> > together
> > > to
> > > > > push this forward.
> > > > >
> > > > > Thanks,
> > > > > Yueting Chen
> > > > >
> > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <dianfu@apache.org>
> wrote:
> > > > >
> > > > > > Hi Fabian,
> > > > > >
> > > > > > We have evaluated the missing features of Flink CEP roughly,
it
> > > should
> > > > > not
> > > > > > be quite difficult to support them. Kostas, Dawid, what's your
> > > thought?
> > > > > >
> > > > > > For supporting MATCH_RECOGNIZE, do you think we could create
the
> > > JIRAs
> > > > > and
> > > > > > start to work right now or we should wait until the release
of
> > > calcite
> > > > > > 1.13?
> > > > > >
> > > > > > Btw, could you help to add me(dian.fu) to the contributor list,
> > then
> > > I
> > > > > can
> > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > >
> > > > > > Best regards,
> > > > > > Dian
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> fhueske@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Jark,
> > > > > > >
> > > > > > > Thanks for updating the design doc and sharing your prototype!
> > > > > > > I didn't look at the code in detail, but the fact that
it is
> less
> > > > than
> > > > > 1k
> > > > > > > LOC is very promising. It seems that most of the complex
CEP
> > logic
> > > > can
> > > > > be
> > > > > > > reused :-)
> > > > > > > Adding a dependency on flink-cep should be fine, IMO. It
is a
> > very
> > > > slim
> > > > > > > library with almost none external dependencies.
> > > > > > >
> > > > > > > Regarding the missing features of Flink CEP that you listed
in
> > the
> > > > > design
> > > > > > > doc, it would be good to get some in put from Kostas and
Dawid
> > > which
> > > > > are
> > > > > > > the main contributors to CEP.
> > > > > > > Do you have already plans regarding some of the missing
> features
> > or
> > > > can
> > > > > > you
> > > > > > > assess how hard it would be to integrate them?
> > > > > > >
> > > > > > > Cheers, Fabian
> > > > > > >
> > > > > > > Btw. The Calcite community started a discussion about releasing
> > > > Calcite
> > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > >
> > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <jark@apache.org>:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > Good news! We have made a prototype for integrating
CEP and
> > SQL.
> > > > See
> > > > > > this
> > > > > > > > link
> > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > >
> > > > > > > >
> > > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > > >
> > > > > > > > Meanwhile, we updated our design doc with additional
> > > implementation
> > > > > > > detail,
> > > > > > > > including how
> > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the
features
> > > needed
> > > > to
> > > > > > add
> > > > > > > > to Flink CEP,
> > > > > > > > and the implementation plan. See the document
> > > > > > > > https://docs.google.com/document/d/
> > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > >
> > > > > > > > In the prototype, we make flink-table dependency on
> flink-cep.
> > Do
> > > > you
> > > > > > > think
> > > > > > > > that is fine?
> > > > > > > >
> > > > > > > > What do you think about the prototype and the design
doc ?
> > > > > > > >
> > > > > > > > Any feedbacks are welcome!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jark Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> trohrmann@apache.org
> > >:
> > > > > > > >
> > > > > > > > > Thanks for sharing your ideas with the community.
I really
> > like
> > > > the
> > > > > > > > design
> > > > > > > > > document and think that it's a good approach
to follow
> > Oracle's
> > > > SQL
> > > > > > > > > extension for pattern matching. Looking forward
to having
> > > support
> > > > > for
> > > > > > > SQL
> > > > > > > > > with CEP capabilities :-)
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Till
> > > > > > > > >
> > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <jark@apache.org>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your
support.
> > > > > > > > > >
> > > > > > > > > > @Fabian, I totally agree with you that we
should focus on
> > SQL
> > > > > > first.
> > > > > > > > > Let's
> > > > > > > > > > keep Table API in mind and discuss that
later.
> > > > > > > > > >
> > > > > > > > > > Regarding to the orderBy() clause, I'm not
sure about
> > that. I
> > > > > think
> > > > > > > it
> > > > > > > > > > makes sense to make it required in streaming
mode(either
> > > order
> > > > by
> > > > > > > > rowtime
> > > > > > > > > > or order by proctime). But CEP also works
in batch mode,
> > and
> > > > not
> > > > > > > > > necessary
> > > > > > > > > > to order by some column. Nevertheless, we
can support CEP
> > on
> > > > > batch
> > > > > > > SQL
> > > > > > > > > > later.
> > > > > > > > > >
> > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE
with
> CEP
> > > > > library
> > > > > > > > (with
> > > > > > > > > > NFA, CEP operator). And we will output a
detailed doc
> and a
> > > > > > prototype
> > > > > > > > in
> > > > > > > > > > the next days.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Jark Wu
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske
<
> > fhueske@gmail.com
> > > >:
> > > > > > > > > >
> > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > >>
> > > > > > > > > >> As you wrote, Till and I (and Kostas)
have been thinking
> > > about
> > > > > > this
> > > > > > > > for
> > > > > > > > > >> some time but haven't had time to work
on this feature.
> > > > > > > > > >> I think it would be a great addition
and value add for
> > > Flink's
> > > > > SQL
> > > > > > > > > >> support and Table API.
> > > > > > > > > >>
> > > > > > > > > >> I read the proposal and think it is
very good. We might
> > need
> > > > to
> > > > > > add
> > > > > > > a
> > > > > > > > > bit
> > > > > > > > > >> more details, esp. when planning the
concrete steps of
> the
> > > > > > > > > implementation.
> > > > > > > > > >>
> > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > >> - IMO, the development should start
focusing on SQL and
> > its
> > > > > > > semantics.
> > > > > > > > > >> Pattern support for the Table API should
be added later.
> > We
> > > > > > followed
> > > > > > > > > that
> > > > > > > > > >> approach for the OVER windows and I
think it worked
> quiet
> > > > well.
> > > > > > > > > >> - We probably want to reuse as much
as possible from the
> > CEP
> > > > > > > library.
> > > > > > > > > >> That means we need to check if the semantics
of the CEP
> > > > library
> > > > > > and
> > > > > > > > > >> Oracle's PATTERN syntax are aligned
(or how we can
> express
> > > the
> > > > > > > PATTERN
> > > > > > > > > >> semantics with the CEP library). This
should be one of
> the
> > > > first
> > > > > > > > steps,
> > > > > > > > > IMO.
> > > > > > > > > >> - I would make the orderBy() clause
required. In regular
> > SQL
> > > > > rows
> > > > > > > have
> > > > > > > > > no
> > > > > > > > > >> order, so we need to make that explicit
(this would also
> > be
> > > > > > > consistent
> > > > > > > > > with
> > > > > > > > > >> the OVER windows).
> > > > > > > > > >>
> > > > > > > > > >> Let me know what you think.
> > > > > > > > > >>
> > > > > > > > > >> Best, Fabian
> > > > > > > > > >>
> > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas
<
> > > > > > > > k.kloudas@data-artisans.com
> > > > > > > > > >:
> > > > > > > > > >>
> > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > >>>
> > > > > > > > > >>> This is a really interesting idea
that has been in our
> > > heads
> > > > > > > > > >>> since the first implementation of
the CEP library.
> > > > > > > > > >>>
> > > > > > > > > >>> A big +1 for moving forward with
this.
> > > > > > > > > >>>
> > > > > > > > > >>> And as for the design document,
I will definitely have
> a
> > > look
> > > > > > > > > >>> and comment there.
> > > > > > > > > >>>
> > > > > > > > > >>> Kostas
> > > > > > > > > >>>
> > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark
Wu <jark@apache.org>
> > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
@Timo, @Till,
> > > @Kostas
> > > > > > > > > >>>
> > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark
Wu <jark@apache.org>:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi devs,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Dian and me and our teammates
have investigated this
> > for a
> > > > > long
> > > > > > > > time.
> > > > > > > > > >>>> We think consolidating Flink
SQL and CEP is an
> exciting
> > > > thing
> > > > > > for
> > > > > > > > > Flink.
> > > > > > > > > >>>> It'll make SQL more powerful
and give users the
> ability
> > to
> > > > > > easily
> > > > > > > > and
> > > > > > > > > >>>> quickly build CEP applications.
 And I find Flink
> > > community
> > > > > has
> > > > > > > also
> > > > > > > > > talked
> > > > > > > > > >>>> about this idea before, such
as the mailing list [1]
> and
> > > [2]
> > > > > and
> > > > > > > > > Fabian &
> > > > > > > > > >>>> Till's talk in Flink Forward
2016 [3].
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think THIS IS THE POINT to
bring up this topic
> again.
> > > > > Because
> > > > > > we
> > > > > > > > > >>>> already have pattern matching
foundation in Flink CEP
> > > > library,
> > > > > > and
> > > > > > > > > Stream
> > > > > > > > > >>>> SQL is ready now and Calcite
has partially supported
> > > pattern
> > > > > > > > matching
> > > > > > > > > >>>> syntax!  We also drafted a design
doc about how to
> > > integrate
> > > > > SQL
> > > > > > > and
> > > > > > > > > CEP,
> > > > > > > > > >>>> and how to support CEP on Table
API.
> > > > > > > https://docs.google.com/docume
> > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > K15H0YbniTnN30/edit?usp=
> > > > > > > sharing
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas
I include you into this
> > > > > > discussion,
> > > > > > > > it
> > > > > > > > > >>>> would be great to hear your
response.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> What do others think?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> [1] http://apache-flink-mailing-
> list-archive.1008284.n3
> > .
> > > > > > nabble.c
> > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > >>>>
> > > > > > > > > >>>> [2] http://apache-flink-mailing-
> list-archive.1008284.n3
> > .
> > > > > > nabble.c
> > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > html#a9790
> > > > > > > > > >>>>
> > > > > > > > > >>>> [3] https://www.slideshare.net/
> tillrohrmann/streaming-
> > > > > > analytics-
> > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Regards,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Jark Wu
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian
Fu <dianfu@apache.org
> >:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Flink's CEP library is a
great library for complex
> > event
> > > > > > > > processing,
> > > > > > > > > >>>>> more
> > > > > > > > > >>>>> and more customers are expressing
their interests in
> > it.
> > > > But
> > > > > it
> > > > > > > > also
> > > > > > > > > >>>>> has
> > > > > > > > > >>>>> some limitations that users
usually have to write a
> lot
> > > of
> > > > > code
> > > > > > > > even
> > > > > > > > > >>>>> for a
> > > > > > > > > >>>>> very simple pattern match
use case as it currently
> only
> > > > > > supports
> > > > > > > > the
> > > > > > > > > >>>>> Java
> > > > > > > > > >>>>> API.
> > > > > > > > > >>>>> We have investigated some
popular CEP products such
> as
> > > > esper
> > > > > > [1]
> > > > > > > > and
> > > > > > > > > >>>>> siddhi
> > > > > > > > > >>>>> [2] and found that most
of these CEP products support
> > > > > SQL-like
> > > > > > > > > >>>>> expressions
> > > > > > > > > >>>>> such as EPL to describe
the match pattern. But these
> > > > > solutions
> > > > > > > also
> > > > > > > > > >>>>> have
> > > > > > > > > >>>>> the drawbacks that the pattern
match languages are
> not
> > > > > standard
> > > > > > > > SQL,
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> learn curve is steep for
users and it's impossible to
> > > > > integrate
> > > > > > > > them
> > > > > > > > > >>>>> into
> > > > > > > > > >>>>> the Flink Table API &
SQL.
> > > > > > > > > >>>>> We find that Oracle's CEP
solution CQL [3] supports a
> > new
> > > > > > pattern
> > > > > > > > > >>>>> recognition clause match_recognize
which is a pattern
> > > > > > recognition
> > > > > > > > > >>>>> clause
> > > > > > > > > >>>>> proposed in this paper [4].
It proposes a set of new
> > > > syntaxes
> > > > > > to
> > > > > > > > > define
> > > > > > > > > >>>>> match pattern in sql expression.
Calcite already
> > supports
> > > > > part
> > > > > > of
> > > > > > > > > this
> > > > > > > > > >>>>> standard [5].  I think it
will be of great value to
> > > support
> > > > > > > > > expressing
> > > > > > > > > >>>>> pattern recognition clause
with match_recognize
> clause
> > by
> > > > > > > > integrating
> > > > > > > > > >>>>> it
> > > > > > > > > >>>>> with Flink Table API &
SQL and the Flink CEP library.
> > Any
> > > > > > > thoughts?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > >>>>> [3]
> > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > eventprocessing/cql-
> > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > CQLLR1531
> > > > > > > > > >>>>> [4]
> > > > > > > > > >>>>> http://web.cs.ucla.edu/
> classes/winter17/cs240B/notes/
> > > > row-pat
> > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > >>>>> [5] https://issues.apache.org/
> jira/browse/CALCITE-1570
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Best Regards,
> > > > > > > > > >>>>> Dian
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message