flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lehuede sebastien <lehued...@gmail.com>
Subject Process stream multiple time with different KeyBy
Date Mon, 17 Feb 2020 18:06:33 GMT
Hi all,

I'm currently working on a Flink Application where I match events against a
set of rules. At the beginning I wanted to dynamically create streams
following the category of events (Event are JSON formatted and I've a field
like "category":"foo" in each event) but I'm stuck by the impossibility to
create streams at runtime.

So, one of the solution for me is to create a single Kafka topic and then
use the "KeyBy" operator to match events with "category":"foo" against
rules also containing "category":"foo" in rule specification.

Now I have some cases where events and rules have one category and one
subcategory. At this point I'm not sure about the "KeyBy" operator behavior.

Example :

   - Events have : "category":"foo" AND "subcategory":"bar"
   - Rule1 specification has : "category":"foo" AND "subcategory":"bar"
   - Rule2 specification has : "category':"foo"
   - Rule3 specification has : "category":"bar"

In this case, my events need to be match against Rule1, Rule2 and Rule3.

If I'm right, if I apply a multiple key "KeyBy()" with "category" and
"subcategory" fields and then apply two single key "KeyBy()" with
"category" field, my events will be consumed by the first "KeyBy()"
operator and no events will be streamed in the operators after ?

Is there any way to process the same stream one time for multi key KeyBy()
and another time for single key KeyBy() ?

Thanks !
S├ębastien.

Mime
View raw message