flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: What happened if my parallelism more than kafka partitions.
Date Wed, 08 Nov 2017 09:37:48 GMT

You can set the parallelism of the Flink Kafka Consumer independent of the number of partitions.
If there are more consumer subtasks than the number of Kafka partitions to read (i.e. when
the parallelism of the consumer is set higher than the number of partitions), some subtasks
will simply remain idle.
Each Kafka partition is deterministically assigned to a single consumer subtask.


On 8 November 2017 at 4:21:54 PM, yunfan123 (yunfanfighting@foxmail.com) wrote:

It seems the same partition data will be consume multi times? 

Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ 

View raw message