Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6BEA7200CB0 for ; Fri, 23 Jun 2017 14:51:40 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6A3D5160BD4; Fri, 23 Jun 2017 12:51:40 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 872FF160BCA for ; Fri, 23 Jun 2017 14:51:39 +0200 (CEST) Received: (qmail 13222 invoked by uid 500); 23 Jun 2017 12:51:33 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 13213 invoked by uid 99); 23 Jun 2017 12:51:33 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jun 2017 12:51:33 +0000 Received: from mail-yw0-f169.google.com (mail-yw0-f169.google.com [209.85.161.169]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 433061A04D1 for ; Fri, 23 Jun 2017 12:51:33 +0000 (UTC) Received: by mail-yw0-f169.google.com with SMTP id 63so16661747ywr.0 for ; Fri, 23 Jun 2017 05:51:33 -0700 (PDT) X-Gm-Message-State: AKS2vOxYLSCZf7bbNYZltix9Ycp46+hPqiyOtQ14abKsS9YJXwxvVIwi yJ2/6vB9hI3NcJLygUe6zR1w9N4ZMQ== X-Received: by 10.129.43.68 with SMTP id r65mr5523206ywr.24.1498222291675; Fri, 23 Jun 2017 05:51:31 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.195.196 with HTTP; Fri, 23 Jun 2017 05:51:31 -0700 (PDT) In-Reply-To: References: <905E90AB-3544-4311-92A0-493BE58DFB0C@gmail.com> <5B9DE3CE-E0DD-47E2-A1CC-1A8DAE6A6BDE@data-artisans.com> From: Suneel Marthi Date: Fri, 23 Jun 2017 08:51:31 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Integrating Flink CEP with a Rules Engine To: =?UTF-8?B?SXNtYcOrbCBNZWrDrWE=?= Cc: Kostas Kloudas , =?UTF-8?B?SsO2cm4gRnJhbmtl?= , Sridhar Chellappa , user@flink.apache.org, Thomas Weise Content-Type: multipart/alternative; boundary="001a1141e7987dc6800552a00f94" archived-at: Fri, 23 Jun 2017 12:51:40 -0000 --001a1141e7987dc6800552a00f94 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable FWIW, here's an old Cloudera blog about using Drools with Spark. https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-process= ing-app-on-apache-spark-and-drools/ It should be possible to invoke Drools from Flink in a similar way (I have not tried it). It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines. Disclaimer: I work for Red Hat On Fri, Jun 23, 2017 at 8:43 AM, Isma=C3=ABl Mej=C3=ADa = wrote: > Hello, > > It is really interesting to see this discussion because that was one > of the questions on the presentation on CEP at Berlin Buzzwords, and > this is one line of work that may eventually make sense to explore. > > Rule engines like drools implement the Rete algorithm that if I > understood correctly optimizes the analysis of a relatively big set of > facts (conditions) into a simpler evaluation graph. For more details > this is a really nice explanation. > https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/ > > On flink's CEP I have the impression that you define this graph by > hand. Using a rule engine you could infer an optimal graph from the > set of rules, and then this graph could be translated into CEP > patterns. > > Of course take all of this with a grain of salt because I am not an > expert on both CEP or the Rete algorithm, but I start to see the > connection of both worlds more clearly now. So if anyone else has > ideas of the feasibility of this or can see some other > issues/consequences please comment. I also have the impression that > distribution is less of an issue because the rete network is > calculated only once and updates are not 'dynamic' (but I might be > wrong). > > Isma=C3=ABl > > ps. I add Thomas in copy who was who made the question in the > conference in case he has some comments/ideas. > > > On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas > wrote: > > Hi Jorn and Sridhar, > > > > It would be worth describing a bit more what these tools are and what a= re > > your needs. > > In addition, and to see what the CEP library already offers here you ca= n > > find the documentation: > > > > https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/libs/cep.html > > > > > > Thanks, > > Kostas > > > > On Jun 23, 2017, at 1:41 PM, J=C3=B6rn Franke wr= ote: > > > > Hallo, > > > > It si possible, but some caveat : flink is a distributed system, but in > > drools the fact are only locally available. This may lead to strange > effects > > when rules update the fact base. > > > > Best regards > > > > On 23. Jun 2017, at 12:49, Sridhar Chellappa > wrote: > > > > Folks, > > > > I am new to Flink. > > > > One of the reasons why I am interested in Flink is because of its CEP > > library. Our CEP logic comprises of a set of complex business rules whi= ch > > will have to be managed (Create, Update, Delete) by a bunch of business > > analysts. > > > > Is there a way I can integrate other third party tools (Drools, > OpenRules) > > to let Business Analysts define rules and execute them using Flink's C= EP > > library? > > > > > --001a1141e7987dc6800552a00f94 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
FWIW, here's an old Cloudera blog about using Drools w= ith Spark.


It = should be possible to invoke Drools from Flink in a similar way (I have not= tried it).=C2=A0

It all depends on what the use c= ase and how much of present Flink CEP satisfies the use case before conside= ring integration with more complex rule engines.

<= br>
Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ism= a=C3=ABl Mej=C3=ADa <iemejia@gmail.com> wrote:
Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com= /rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be<= br> wrong).

Isma=C3=ABl

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.


On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<k.kloudas@data-artisans.= com> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what = are
> your needs.
> In addition, and to see what the CEP library already offers here you c= an
> find the documentation:
>
> https://ci.apache.= org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html >
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, J=C3=B6rn Franke <jornfranke@gmail.com> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but i= n
> drools the fact are only locally available. This may lead to strange e= ffects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <flinkenthu@gmail.com> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP<= br> > library. Our CEP logic comprises of a set of complex business rules wh= ich
> will have to be managed (Create, Update, Delete) by a bunch of busines= s
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRu= les)
> to let Business Analysts define rules and=C2=A0 execute them using Fli= nk's CEP
> library?
>
>

--001a1141e7987dc6800552a00f94--