flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taher Koitawala <taher.koitaw...@gslab.com>
Subject Re: understadning kafka connector - rebalance
Date Mon, 26 Nov 2018 06:33:01 GMT
Hi Avi,
          No, rebalance is not changing the number of kafka partitions.
Lets say you have 6 kafka partitions and your flink parallelism is 8, in
this case using rebalance will send records to all downstream operators in
a round robin fashion.

Taher Koitawala
GS Lab Pune
+91 8407979163

On Mon, Nov 26, 2018 at 11:33 AM Avi Levi <avi.levi@bluevoyant.com> wrote:

> Hi
> Looking at this example
> <https://github.com/dataArtisans/kafka-example/blob/master/src/main/java/com/dataartisans/ReadFromKafka.java>,
> doing the "rebalance" (e.g messageStream.rebalance().map(...) ) operation
> on heavy load stream wouldn't slow the stream ? is the rebalancing action
> occurs only when there is a partition change ?
> it says that "the rebelance call is causing a repartitioning of the data
> so that all machines" is it actually changing the num of partitions of
> the topic to match the num of flink operators ?
> Avi

View raw message