flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CPC <acha...@gmail.com>
Subject windowAll and AggregateFunction
Date Wed, 09 Jan 2019 12:43:52 GMT
Hi all,

In our implementation,we are consuming from kafka and calculating distinct
with hyperloglog. We are using windowAll function with a custom
AggregateFunction but flink runtime shows a little bit unexpected behavior
at runtime. Our sources running with parallelism 4 and i expect add
function to run after source calculate partial results and at the end of
the window i expect it to send 4 hll object to single operator to merge
there(merge function). Instead, it sends all data to single instance and
call add function there.

Is here any way to make flink behave like this? I mean calculate partial
results after consuming from kafka with paralelism of sources without
shuffling(so some part of the calculation can be calculated in parallel)
and merge those partial results with a merge function?

Thank you in advance...

View raw message