flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jark Wu <j...@apache.org>
Subject Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP
Date Tue, 13 Jun 2017 07:34:36 GMT
Thank you Yueting for pointing out the mistake in the prototype. I
accidentally introduced it when merge code.

I'm so glad to see so many people are interested in the feature. Let's work
out together to push it forward!

Cheers,
Jark Wu


2017-06-13 15:27 GMT+08:00 Liangfei Su <suliangfei@gmail.com>:

> +1 for the feature. Myself was a user of Siddhi, this is pretty user
> friendly feature to provide to user.
>
> On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <dian0511.fu@gmail.com> wrote:
>
> > Hi Yueting & Dawid Wysakowicz,
> >
> > Very glad that you're interested in this feature and you're definitely
> > welcome to join this work and also anyone else.:)
> >
> > Best regards,
> > Dian
> >
> > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > wysakowicz.dawid@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Integrating SQL with CEP seems like a really nice idea. Unfortunately I
> > had
> > > time just for a brief look at the design doc, but it looks really cool
> > and
> > > thorough. Also will have a second run tomorrow and will try to provide
> > more
> > > comments. Anyway will be glad to help pushing the initiative forward.
> > >
> > > Z pozdrowieniami! / Cheers!
> > >
> > > Dawid Wysakowicz
> > >
> > > *Data/Software Engineer*
> > >
> > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > >
> > > <http://getindata.com/>
> > >
> > > 2017-06-13 8:19 GMT+02:00 yueting chen <yestinmail@gmail.com>:
> > >
> > > > Hi Dian & Jark,
> > > >
> > > > I checked out your prototype code, but it didn't pass the CEPITCase
> > test
> > > in
> > > > the flink-table component.
> > > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> > should
> > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > instead
> > > > of
> > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > >
> > > > I've also read your desgin document and it looks fine to me.
> Actually,
> > I
> > > am
> > > > working on the same thing recently, I think maybe we can work
> together
> > to
> > > > push this forward.
> > > >
> > > > Thanks,
> > > > Yueting Chen
> > > >
> > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <dianfu@apache.org> wrote:
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > We have evaluated the missing features of Flink CEP roughly, it
> > should
> > > > not
> > > > > be quite difficult to support them. Kostas, Dawid, what's your
> > thought?
> > > > >
> > > > > For supporting MATCH_RECOGNIZE, do you think we could create the
> > JIRAs
> > > > and
> > > > > start to work right now or we should wait until the release of
> > calcite
> > > > > 1.13?
> > > > >
> > > > > Btw, could you help to add me(dian.fu) to the contributor list,
> then
> > I
> > > > can
> > > > > assign the JIRAs to myself? Thanks a lot.
> > > > >
> > > > > Best regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <fhueske@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Jark,
> > > > > >
> > > > > > Thanks for updating the design doc and sharing your prototype!
> > > > > > I didn't look at the code in detail, but the fact that it is
less
> > > than
> > > > 1k
> > > > > > LOC is very promising. It seems that most of the complex CEP
> logic
> > > can
> > > > be
> > > > > > reused :-)
> > > > > > Adding a dependency on flink-cep should be fine, IMO. It is
a
> very
> > > slim
> > > > > > library with almost none external dependencies.
> > > > > >
> > > > > > Regarding the missing features of Flink CEP that you listed
in
> the
> > > > design
> > > > > > doc, it would be good to get some in put from Kostas and Dawid
> > which
> > > > are
> > > > > > the main contributors to CEP.
> > > > > > Do you have already plans regarding some of the missing features
> or
> > > can
> > > > > you
> > > > > > assess how hard it would be to integrate them?
> > > > > >
> > > > > > Cheers, Fabian
> > > > > >
> > > > > > Btw. The Calcite community started a discussion about releasing
> > > Calcite
> > > > > > 1.13. So, the missing features might soon be available.
> > > > > >
> > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <jark@apache.org>:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > Good news! We have made a prototype for integrating CEP
and
> SQL.
> > > See
> > > > > this
> > > > > > > link
> > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > >
> > > > > > >
> > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > >
> > > > > > > Meanwhile, we updated our design doc with additional
> > implementation
> > > > > > detail,
> > > > > > > including how
> > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the features
> > needed
> > > to
> > > > > add
> > > > > > > to Flink CEP,
> > > > > > > and the implementation plan. See the document
> > > > > > > https://docs.google.com/document/d/
> > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > >
> > > > > > > In the prototype, we make flink-table dependency on flink-cep.
> Do
> > > you
> > > > > > think
> > > > > > > that is fine?
> > > > > > >
> > > > > > > What do you think about the prototype and the design doc
?
> > > > > > >
> > > > > > > Any feedbacks are welcome!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jark Wu
> > > > > > >
> > > > > > >
> > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <trohrmann@apache.org
> >:
> > > > > > >
> > > > > > > > Thanks for sharing your ideas with the community.
I really
> like
> > > the
> > > > > > > design
> > > > > > > > document and think that it's a good approach to follow
> Oracle's
> > > SQL
> > > > > > > > extension for pattern matching. Looking forward to
having
> > support
> > > > for
> > > > > > SQL
> > > > > > > > with CEP capabilities :-)
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <jark@apache.org>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > >
> > > > > > > > > @Fabian, I totally agree with you that we should
focus on
> SQL
> > > > > first.
> > > > > > > > Let's
> > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > >
> > > > > > > > > Regarding to the orderBy() clause, I'm not sure
about
> that. I
> > > > think
> > > > > > it
> > > > > > > > > makes sense to make it required in streaming
mode(either
> > order
> > > by
> > > > > > > rowtime
> > > > > > > > > or order by proctime). But CEP also works in
batch mode,
> and
> > > not
> > > > > > > > necessary
> > > > > > > > > to order by some column. Nevertheless, we can
support CEP
> on
> > > > batch
> > > > > > SQL
> > > > > > > > > later.
> > > > > > > > >
> > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE
with CEP
> > > > library
> > > > > > > (with
> > > > > > > > > NFA, CEP operator). And we will output a detailed
doc and a
> > > > > prototype
> > > > > > > in
> > > > > > > > > the next days.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> fhueske@gmail.com
> > >:
> > > > > > > > >
> > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > >>
> > > > > > > > >> As you wrote, Till and I (and Kostas) have
been thinking
> > about
> > > > > this
> > > > > > > for
> > > > > > > > >> some time but haven't had time to work on
this feature.
> > > > > > > > >> I think it would be a great addition and
value add for
> > Flink's
> > > > SQL
> > > > > > > > >> support and Table API.
> > > > > > > > >>
> > > > > > > > >> I read the proposal and think it is very
good. We might
> need
> > > to
> > > > > add
> > > > > > a
> > > > > > > > bit
> > > > > > > > >> more details, esp. when planning the concrete
steps of the
> > > > > > > > implementation.
> > > > > > > > >>
> > > > > > > > >> A few comments to the proposal:
> > > > > > > > >> - IMO, the development should start focusing
on SQL and
> its
> > > > > > semantics.
> > > > > > > > >> Pattern support for the Table API should
be added later.
> We
> > > > > followed
> > > > > > > > that
> > > > > > > > >> approach for the OVER windows and I think
it worked quiet
> > > well.
> > > > > > > > >> - We probably want to reuse as much as possible
from the
> CEP
> > > > > > library.
> > > > > > > > >> That means we need to check if the semantics
of the CEP
> > > library
> > > > > and
> > > > > > > > >> Oracle's PATTERN syntax are aligned (or how
we can express
> > the
> > > > > > PATTERN
> > > > > > > > >> semantics with the CEP library). This should
be one of the
> > > first
> > > > > > > steps,
> > > > > > > > IMO.
> > > > > > > > >> - I would make the orderBy() clause required.
In regular
> SQL
> > > > rows
> > > > > > have
> > > > > > > > no
> > > > > > > > >> order, so we need to make that explicit (this
would also
> be
> > > > > > consistent
> > > > > > > > with
> > > > > > > > >> the OVER windows).
> > > > > > > > >>
> > > > > > > > >> Let me know what you think.
> > > > > > > > >>
> > > > > > > > >> Best, Fabian
> > > > > > > > >>
> > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas
<
> > > > > > > k.kloudas@data-artisans.com
> > > > > > > > >:
> > > > > > > > >>
> > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > >>>
> > > > > > > > >>> This is a really interesting idea that
has been in our
> > heads
> > > > > > > > >>> since the first implementation of the
CEP library.
> > > > > > > > >>>
> > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > >>>
> > > > > > > > >>> And as for the design document, I will
definitely have a
> > look
> > > > > > > > >>> and comment there.
> > > > > > > > >>>
> > > > > > > > >>> Kostas
> > > > > > > > >>>
> > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu
<jark@apache.org>
> > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
@Timo, @Till,
> > @Kostas
> > > > > > > > >>>
> > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <jark@apache.org>:
> > > > > > > > >>>
> > > > > > > > >>>> Hi devs,
> > > > > > > > >>>>
> > > > > > > > >>>> Dian and me and our teammates have
investigated this
> for a
> > > > long
> > > > > > > time.
> > > > > > > > >>>> We think consolidating Flink SQL
and CEP is an exciting
> > > thing
> > > > > for
> > > > > > > > Flink.
> > > > > > > > >>>> It'll make SQL more powerful and
give users the ability
> to
> > > > > easily
> > > > > > > and
> > > > > > > > >>>> quickly build CEP applications. 
And I find Flink
> > community
> > > > has
> > > > > > also
> > > > > > > > talked
> > > > > > > > >>>> about this idea before, such as the
mailing list [1] and
> > [2]
> > > > and
> > > > > > > > Fabian &
> > > > > > > > >>>> Till's talk in Flink Forward 2016
[3].
> > > > > > > > >>>>
> > > > > > > > >>>> I think THIS IS THE POINT to bring
up this topic again.
> > > > Because
> > > > > we
> > > > > > > > >>>> already have pattern matching foundation
in Flink CEP
> > > library,
> > > > > and
> > > > > > > > Stream
> > > > > > > > >>>> SQL is ready now and Calcite has
partially supported
> > pattern
> > > > > > > matching
> > > > > > > > >>>> syntax!  We also drafted a design
doc about how to
> > integrate
> > > > SQL
> > > > > > and
> > > > > > > > CEP,
> > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > https://docs.google.com/docume
> > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > K15H0YbniTnN30/edit?usp=
> > > > > > sharing
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I
include you into this
> > > > > discussion,
> > > > > > > it
> > > > > > > > >>>> would be great to hear your response.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> What do others think?
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3
> .
> > > > > nabble.c
> > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > >>>>
> > > > > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3
> .
> > > > > nabble.c
> > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> html#a9790
> > > > > > > > >>>>
> > > > > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> > > > > analytics-
> > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>>
> > > > > > > > >>>> Jark Wu
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu
<dianfu@apache.org>:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hi everyone,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Flink's CEP library is a great
library for complex
> event
> > > > > > > processing,
> > > > > > > > >>>>> more
> > > > > > > > >>>>> and more customers are expressing
their interests in
> it.
> > > But
> > > > it
> > > > > > > also
> > > > > > > > >>>>> has
> > > > > > > > >>>>> some limitations that users usually
have to write a lot
> > of
> > > > code
> > > > > > > even
> > > > > > > > >>>>> for a
> > > > > > > > >>>>> very simple pattern match use
case as it currently only
> > > > > supports
> > > > > > > the
> > > > > > > > >>>>> Java
> > > > > > > > >>>>> API.
> > > > > > > > >>>>> We have investigated some popular
CEP products such as
> > > esper
> > > > > [1]
> > > > > > > and
> > > > > > > > >>>>> siddhi
> > > > > > > > >>>>> [2] and found that most of these
CEP products support
> > > > SQL-like
> > > > > > > > >>>>> expressions
> > > > > > > > >>>>> such as EPL to describe the match
pattern. But these
> > > > solutions
> > > > > > also
> > > > > > > > >>>>> have
> > > > > > > > >>>>> the drawbacks that the pattern
match languages are not
> > > > standard
> > > > > > > SQL,
> > > > > > > > >>>>> the
> > > > > > > > >>>>> learn curve is steep for users
and it's impossible to
> > > > integrate
> > > > > > > them
> > > > > > > > >>>>> into
> > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > >>>>> We find that Oracle's CEP solution
CQL [3] supports a
> new
> > > > > pattern
> > > > > > > > >>>>> recognition clause match_recognize
which is a pattern
> > > > > recognition
> > > > > > > > >>>>> clause
> > > > > > > > >>>>> proposed in this paper [4]. It
proposes a set of new
> > > syntaxes
> > > > > to
> > > > > > > > define
> > > > > > > > >>>>> match pattern in sql expression.
Calcite already
> supports
> > > > part
> > > > > of
> > > > > > > > this
> > > > > > > > >>>>> standard [5].  I think it will
be of great value to
> > support
> > > > > > > > expressing
> > > > > > > > >>>>> pattern recognition clause with
match_recognize clause
> by
> > > > > > > integrating
> > > > > > > > >>>>> it
> > > > > > > > >>>>> with Flink Table API & SQL
and the Flink CEP library.
> Any
> > > > > > thoughts?
> > > > > > > > >>>>>
> > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > >>>>> [3]
> > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > eventprocessing/cql-
> > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> 4BC7-B1CE-C84B202217BD.htm#
> > > > > > CQLLR1531
> > > > > > > > >>>>> [4]
> > > > > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/
> > > row-pat
> > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > > > > > >>>>>
> > > > > > > > >>>>> Best Regards,
> > > > > > > > >>>>> Dian
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message