flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maciek Próchniak <...@touk.pl>
Subject Re: Advice - Drools in Flink
Date Wed, 22 Jun 2016 19:00:20 GMT

I've been using drools in a few projects. Embedding Drools in Flink is 
certainly possible, however there are a few points you have to consider.
- are you going to use Drools in stateful or stateless way? The blog 
post you mention uses flink in stateless way - that is, each event is 
processed separately and
each drools session behaves like a bunch of if's. This is 
straightforward, it also shouldn't be problematic performance wise
OTOH, if you want to use stateful sessions things are getting more 
complicated - because if you want to play well with Flink you'd have to 
integrate drools sessions with checkpoints in Flink - and that can be a 
bit more tricky (although certainly possible) - especially when it comes 
to size of session, partitioning and so on.
Even more difficult would be to use Drools CEP features - because then 
you'd have to consider how to handle time (e.g. during restore??).
To be honest, I wouldn't try to integrate Drools CEP with Flink - Flink 
has it's own time handling, own CEP engine and mixing those two can lead 
to tricky issues...

Even if you decide to use stateless sessions there's issue with 
deployment. If you don't want to redeploy Flink process each time one of 
the users change rules, you'd have to implement some notion of detecting 
changes (and storing rules somewhere?) and recompiling rules on each 
task manager. Again - this is certainly doable and possible, but 
requires some thinking about the design.

To sum up - there are many ways you can integrate drools with flink. I'd 
start with something simple - exactly as Robert wrote
hope this helps a bit and good luck,


On Tue, Jun 21, 2016 at 7:55 AM, Anton <kurrent93@gmail.com> wrote:

> Hello
> Firstly, I am an absolute Flink newbie.
> I am interested in using Drools in Flink - in a similar case to what is
> described in this blog, where Drools is used in Spark.
> http://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/
> The basic idea is that Drools can be used to reason over streaming data.
> The high-level use case is, there are several hundred users who want to
> write rules to be notified on events related to changes in specific
> streams. For example, notify me when a specific stock price changes by so
> much.
> Due to the number of users, the more end-user focused syntax of Drools, and
> the number of rule changes, doing this in Drools makes more sense than to
> write and deploy plain-old-flink (apologies, am not familiar with the
> correct term for a flink process).
> Also, as Drools has some powerful CEP operators, it could be very
> interested to have these available in Flink too.
> My question, therefore, is, very general - how best to integrate Drools
> into Flink? Where should I start?
> Thanks and regards
> Anton

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message