flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Best Practices/Advice - Execution of jobs
Date Thu, 03 Nov 2016 15:01:29 GMT
Hi Pedro,
I think it would be better to have two jobs and keep all the rules in one
place. If it's not too many sources you might even consider having
everything in one job so you don't have to duplicate the rules.

There's a tradeoff, though, if it becomes too much stuff then splitting up
will be beneficial because it makes the jobs easier to maintain/monitor.

Cheers,
Aljoscha

On Wed, 2 Nov 2016 at 10:26 PedroMrChaves <pedro.mr.chaves@gmail.com> wrote:

> Hello,
>
> I'm trying to build a stream event correlation engine with Flink and I have
> some questions regarding the for the execution of jobs.
>
> In my architecture I need to have different sources of data, lets say for
> instance:
> /firewallStream= environment.addSource([FirewalLogsSource]);
> proxyStream = environment.addSource([ProxyLogsSource]);
> /
> and for each of these sources, I need to apply a set of rules.
> So lets say I have a job that has as a source the proxy stream data with
> the
> following rules:
>
> ///Abnormal Request Method
> stream.[RuleLogic].addSink([output])
> //Web Service on Non-Typical Port
> stream.[RuleLogic].addSink([output])
> //Possible Brute Force
> stream.[RuleLogic].addSink([output])/
>
> These rules will probably scale to be in the order of 15 to 20 rules.
>
> What is the best approach in this case:
> 1. Should I create 2 jobs one for each source and each job would have the
> 15-20 rules?
> 2. Should I split the rules into several jobs?
> 3. Other options?
>
>
> Thank you and Regards,
> Pedro Chaves.
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-Practices-Advice-Execution-of-jobs-tp9822.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Mime
View raw message