flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP
Date Tue, 13 Jun 2017 08:26:18 GMT
Hi,

@Dian Fu: I gave you contributor permissions. :-)

I don't think we have to wait for Calcite 1.13 to start working on the
missing features of the CEP library and extending the prototype.
We might want to wait with the integration into flink-table until Calcite
1.13 is out and we updated the dependency though.

Best, Fabian

2017-06-13 9:45 GMT+02:00 jincheng sun <sunjincheng121@gmail.com>:

> Hi Jark, Dian,
>
> Thanks for bring up this discuss and share the prototype.
>
> +1 to push this great feature forward!
>
> Cheers,
> SunJincheng
>
> 2017-06-13 15:34 GMT+08:00 Jark Wu <jark@apache.org>:
>
> > Thank you Yueting for pointing out the mistake in the prototype. I
> > accidentally introduced it when merge code.
> >
> > I'm so glad to see so many people are interested in the feature. Let's
> work
> > out together to push it forward!
> >
> > Cheers,
> > Jark Wu
> >
> >
> > 2017-06-13 15:27 GMT+08:00 Liangfei Su <suliangfei@gmail.com>:
> >
> > > +1 for the feature. Myself was a user of Siddhi, this is pretty user
> > > friendly feature to provide to user.
> > >
> > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <dian0511.fu@gmail.com>
> wrote:
> > >
> > > > Hi Yueting & Dawid Wysakowicz,
> > > >
> > > > Very glad that you're interested in this feature and you're
> definitely
> > > > welcome to join this work and also anyone else.:)
> > > >
> > > > Best regards,
> > > > Dian
> > > >
> > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > wysakowicz.dawid@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Integrating SQL with CEP seems like a really nice idea.
> > Unfortunately I
> > > > had
> > > > > time just for a brief look at the design doc, but it looks really
> > cool
> > > > and
> > > > > thorough. Also will have a second run tomorrow and will try to
> > provide
> > > > more
> > > > > comments. Anyway will be glad to help pushing the initiative
> forward.
> > > > >
> > > > > Z pozdrowieniami! / Cheers!
> > > > >
> > > > > Dawid Wysakowicz
> > > > >
> > > > > *Data/Software Engineer*
> > > > >
> > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > >
> > > > > <http://getindata.com/>
> > > > >
> > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <yestinmail@gmail.com>:
> > > > >
> > > > > > Hi Dian & Jark,
> > > > > >
> > > > > > I checked out your prototype code, but it didn't pass the
> CEPITCase
> > > > test
> > > > > in
> > > > > > the flink-table component.
> > > > > > It turns out that in the `MatchCodeGenerator.scala` file, line
74
> > > > should
> > > > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > > > instead
> > > > > > of
> > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > >
> > > > > > I've also read your desgin document and it looks fine to me.
> > > Actually,
> > > > I
> > > > > am
> > > > > > working on the same thing recently, I think maybe we can work
> > > together
> > > > to
> > > > > > push this forward.
> > > > > >
> > > > > > Thanks,
> > > > > > Yueting Chen
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <dianfu@apache.org>
> > wrote:
> > > > > >
> > > > > > > Hi Fabian,
> > > > > > >
> > > > > > > We have evaluated the missing features of Flink CEP roughly,
it
> > > > should
> > > > > > not
> > > > > > > be quite difficult to support them. Kostas, Dawid, what's
your
> > > > thought?
> > > > > > >
> > > > > > > For supporting MATCH_RECOGNIZE, do you think we could create
> the
> > > > JIRAs
> > > > > > and
> > > > > > > start to work right now or we should wait until the release
of
> > > > calcite
> > > > > > > 1.13?
> > > > > > >
> > > > > > > Btw, could you help to add me(dian.fu) to the contributor
list,
> > > then
> > > > I
> > > > > > can
> > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Dian
> > > > > > >
> > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > fhueske@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jark,
> > > > > > > >
> > > > > > > > Thanks for updating the design doc and sharing your
> prototype!
> > > > > > > > I didn't look at the code in detail, but the fact
that it is
> > less
> > > > > than
> > > > > > 1k
> > > > > > > > LOC is very promising. It seems that most of the complex
CEP
> > > logic
> > > > > can
> > > > > > be
> > > > > > > > reused :-)
> > > > > > > > Adding a dependency on flink-cep should be fine, IMO.
It is a
> > > very
> > > > > slim
> > > > > > > > library with almost none external dependencies.
> > > > > > > >
> > > > > > > > Regarding the missing features of Flink CEP that you
listed
> in
> > > the
> > > > > > design
> > > > > > > > doc, it would be good to get some in put from Kostas
and
> Dawid
> > > > which
> > > > > > are
> > > > > > > > the main contributors to CEP.
> > > > > > > > Do you have already plans regarding some of the missing
> > features
> > > or
> > > > > can
> > > > > > > you
> > > > > > > > assess how hard it would be to integrate them?
> > > > > > > >
> > > > > > > > Cheers, Fabian
> > > > > > > >
> > > > > > > > Btw. The Calcite community started a discussion about
> releasing
> > > > > Calcite
> > > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > > >
> > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <jark@apache.org>:
> > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > Good news! We have made a prototype for integrating
CEP and
> > > SQL.
> > > > > See
> > > > > > > this
> > > > > > > > > link
> > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > You can check CepITCase to try the simple CQL
example.
> > > > > > > > >
> > > > > > > > > Meanwhile, we updated our design doc with additional
> > > > implementation
> > > > > > > > detail,
> > > > > > > > > including how
> > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and
the features
> > > > needed
> > > > > to
> > > > > > > add
> > > > > > > > > to Flink CEP,
> > > > > > > > > and the implementation plan. See the document
> > > > > > > > > https://docs.google.com/document/d/
> > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > >
> > > > > > > > > In the prototype, we make flink-table dependency
on
> > flink-cep.
> > > Do
> > > > > you
> > > > > > > > think
> > > > > > > > > that is fine?
> > > > > > > > >
> > > > > > > > > What do you think about the prototype and the
design doc ?
> > > > > > > > >
> > > > > > > > > Any feedbacks are welcome!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > trohrmann@apache.org
> > > >:
> > > > > > > > >
> > > > > > > > > > Thanks for sharing your ideas with the community.
I
> really
> > > like
> > > > > the
> > > > > > > > > design
> > > > > > > > > > document and think that it's a good approach
to follow
> > > Oracle's
> > > > > SQL
> > > > > > > > > > extension for pattern matching. Looking
forward to having
> > > > support
> > > > > > for
> > > > > > > > SQL
> > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Till
> > > > > > > > > >
> > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu
<jark@apache.org
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for
your support.
> > > > > > > > > > >
> > > > > > > > > > > @Fabian, I totally agree with you that
we should focus
> on
> > > SQL
> > > > > > > first.
> > > > > > > > > > Let's
> > > > > > > > > > > keep Table API in mind and discuss
that later.
> > > > > > > > > > >
> > > > > > > > > > > Regarding to the orderBy() clause,
I'm not sure about
> > > that. I
> > > > > > think
> > > > > > > > it
> > > > > > > > > > > makes sense to make it required in
streaming
> mode(either
> > > > order
> > > > > by
> > > > > > > > > rowtime
> > > > > > > > > > > or order by proctime). But CEP also
works in batch
> mode,
> > > and
> > > > > not
> > > > > > > > > > necessary
> > > > > > > > > > > to order by some column. Nevertheless,
we can support
> CEP
> > > on
> > > > > > batch
> > > > > > > > SQL
> > > > > > > > > > > later.
> > > > > > > > > > >
> > > > > > > > > > > We are estimating how to implement
MATCH_RECOGNIZE with
> > CEP
> > > > > > library
> > > > > > > > > (with
> > > > > > > > > > > NFA, CEP operator). And we will output
a detailed doc
> > and a
> > > > > > > prototype
> > > > > > > > > in
> > > > > > > > > > > the next days.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Jark Wu
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske
<
> > > fhueske@gmail.com
> > > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > >>
> > > > > > > > > > >> As you wrote, Till and I (and Kostas)
have been
> thinking
> > > > about
> > > > > > > this
> > > > > > > > > for
> > > > > > > > > > >> some time but haven't had time
to work on this
> feature.
> > > > > > > > > > >> I think it would be a great addition
and value add for
> > > > Flink's
> > > > > > SQL
> > > > > > > > > > >> support and Table API.
> > > > > > > > > > >>
> > > > > > > > > > >> I read the proposal and think it
is very good. We
> might
> > > need
> > > > > to
> > > > > > > add
> > > > > > > > a
> > > > > > > > > > bit
> > > > > > > > > > >> more details, esp. when planning
the concrete steps of
> > the
> > > > > > > > > > implementation.
> > > > > > > > > > >>
> > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > >> - IMO, the development should start
focusing on SQL
> and
> > > its
> > > > > > > > semantics.
> > > > > > > > > > >> Pattern support for the Table API
should be added
> later.
> > > We
> > > > > > > followed
> > > > > > > > > > that
> > > > > > > > > > >> approach for the OVER windows and
I think it worked
> > quiet
> > > > > well.
> > > > > > > > > > >> - We probably want to reuse as
much as possible from
> the
> > > CEP
> > > > > > > > library.
> > > > > > > > > > >> That means we need to check if
the semantics of the
> CEP
> > > > > library
> > > > > > > and
> > > > > > > > > > >> Oracle's PATTERN syntax are aligned
(or how we can
> > express
> > > > the
> > > > > > > > PATTERN
> > > > > > > > > > >> semantics with the CEP library).
This should be one of
> > the
> > > > > first
> > > > > > > > > steps,
> > > > > > > > > > IMO.
> > > > > > > > > > >> - I would make the orderBy() clause
required. In
> regular
> > > SQL
> > > > > > rows
> > > > > > > > have
> > > > > > > > > > no
> > > > > > > > > > >> order, so we need to make that
explicit (this would
> also
> > > be
> > > > > > > > consistent
> > > > > > > > > > with
> > > > > > > > > > >> the OVER windows).
> > > > > > > > > > >>
> > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > >>
> > > > > > > > > > >> Best, Fabian
> > > > > > > > > > >>
> > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas
Kloudas <
> > > > > > > > > k.kloudas@data-artisans.com
> > > > > > > > > > >:
> > > > > > > > > > >>
> > > > > > > > > > >>> Thanks a lot for opening the
discussion!
> > > > > > > > > > >>>
> > > > > > > > > > >>> This is a really interesting
idea that has been in
> our
> > > > heads
> > > > > > > > > > >>> since the first implementation
of the CEP library.
> > > > > > > > > > >>>
> > > > > > > > > > >>> A big +1 for moving forward
with this.
> > > > > > > > > > >>>
> > > > > > > > > > >>> And as for the design document,
I will definitely
> have
> > a
> > > > look
> > > > > > > > > > >>> and comment there.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Kostas
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM,
Jark Wu <
> jark@apache.org>
> > > > > wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> Sorry, I forgot to cc you guys
@Fabian, @Timo, @Till,
> > > > @Kostas
> > > > > > > > > > >>>
> > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00
Jark Wu <jark@apache.org
> >:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Dian and me and our teammates
have investigated this
> > > for a
> > > > > > long
> > > > > > > > > time.
> > > > > > > > > > >>>> We think consolidating
Flink SQL and CEP is an
> > exciting
> > > > > thing
> > > > > > > for
> > > > > > > > > > Flink.
> > > > > > > > > > >>>> It'll make SQL more powerful
and give users the
> > ability
> > > to
> > > > > > > easily
> > > > > > > > > and
> > > > > > > > > > >>>> quickly build CEP applications.
 And I find Flink
> > > > community
> > > > > > has
> > > > > > > > also
> > > > > > > > > > talked
> > > > > > > > > > >>>> about this idea before,
such as the mailing list [1]
> > and
> > > > [2]
> > > > > > and
> > > > > > > > > > Fabian &
> > > > > > > > > > >>>> Till's talk in Flink Forward
2016 [3].
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think THIS IS THE POINT
to bring up this topic
> > again.
> > > > > > Because
> > > > > > > we
> > > > > > > > > > >>>> already have pattern matching
foundation in Flink
> CEP
> > > > > library,
> > > > > > > and
> > > > > > > > > > Stream
> > > > > > > > > > >>>> SQL is ready now and Calcite
has partially supported
> > > > pattern
> > > > > > > > > matching
> > > > > > > > > > >>>> syntax!  We also drafted
a design doc about how to
> > > > integrate
> > > > > > SQL
> > > > > > > > and
> > > > > > > > > > CEP,
> > > > > > > > > > >>>> and how to support CEP
on Table API.
> > > > > > > > https://docs.google.com/docume
> > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > sharing
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> @Fabian, @Timo, @Till,
@Kostas I include you into
> this
> > > > > > > discussion,
> > > > > > > > > it
> > > > > > > > > > >>>> would be great to hear
your response.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> What do others think?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > list-archive.1008284.n3
> > > .
> > > > > > > nabble.c
> > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > list-archive.1008284.n3
> > > .
> > > > > > > nabble.c
> > > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > > html#a9790
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > tillrohrmann/streaming-
> > > > > > > analytics-
> > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Regards,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00
Dian Fu <
> dianfu@apache.org
> > >:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Flink's CEP library
is a great library for complex
> > > event
> > > > > > > > > processing,
> > > > > > > > > > >>>>> more
> > > > > > > > > > >>>>> and more customers
are expressing their interests
> in
> > > it.
> > > > > But
> > > > > > it
> > > > > > > > > also
> > > > > > > > > > >>>>> has
> > > > > > > > > > >>>>> some limitations that
users usually have to write a
> > lot
> > > > of
> > > > > > code
> > > > > > > > > even
> > > > > > > > > > >>>>> for a
> > > > > > > > > > >>>>> very simple pattern
match use case as it currently
> > only
> > > > > > > supports
> > > > > > > > > the
> > > > > > > > > > >>>>> Java
> > > > > > > > > > >>>>> API.
> > > > > > > > > > >>>>> We have investigated
some popular CEP products such
> > as
> > > > > esper
> > > > > > > [1]
> > > > > > > > > and
> > > > > > > > > > >>>>> siddhi
> > > > > > > > > > >>>>> [2] and found that
most of these CEP products
> support
> > > > > > SQL-like
> > > > > > > > > > >>>>> expressions
> > > > > > > > > > >>>>> such as EPL to describe
the match pattern. But
> these
> > > > > > solutions
> > > > > > > > also
> > > > > > > > > > >>>>> have
> > > > > > > > > > >>>>> the drawbacks that
the pattern match languages are
> > not
> > > > > > standard
> > > > > > > > > SQL,
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> learn curve is steep
for users and it's impossible
> to
> > > > > > integrate
> > > > > > > > > them
> > > > > > > > > > >>>>> into
> > > > > > > > > > >>>>> the Flink Table API
& SQL.
> > > > > > > > > > >>>>> We find that Oracle's
CEP solution CQL [3]
> supports a
> > > new
> > > > > > > pattern
> > > > > > > > > > >>>>> recognition clause
match_recognize which is a
> pattern
> > > > > > > recognition
> > > > > > > > > > >>>>> clause
> > > > > > > > > > >>>>> proposed in this paper
[4]. It proposes a set of
> new
> > > > > syntaxes
> > > > > > > to
> > > > > > > > > > define
> > > > > > > > > > >>>>> match pattern in sql
expression. Calcite already
> > > supports
> > > > > > part
> > > > > > > of
> > > > > > > > > > this
> > > > > > > > > > >>>>> standard [5].  I think
it will be of great value to
> > > > support
> > > > > > > > > > expressing
> > > > > > > > > > >>>>> pattern recognition
clause with match_recognize
> > clause
> > > by
> > > > > > > > > integrating
> > > > > > > > > > >>>>> it
> > > > > > > > > > >>>>> with Flink Table API
& SQL and the Flink CEP
> library.
> > > Any
> > > > > > > > thoughts?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > >>>>> [3]
> > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > eventprocessing/cql-
> > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > CQLLR1531
> > > > > > > > > > >>>>> [4]
> > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > classes/winter17/cs240B/notes/
> > > > > row-pat
> > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > jira/browse/CALCITE-1570
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > >>>>> Dian
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message