flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yunfan123 <yunfanfight...@foxmail.com>
Subject Re: What happened if my parallelism more than kafka partitions.
Date Wed, 08 Nov 2017 10:38:49 GMT
The code of kafka partition assign is like follows:

public static int assign(KafkaTopicPartition partition, int
numParallelSubtasks) {
		int startIndex = ((partition.getTopic().hashCode() * 31) & 0x7FFFFFFF) %

		// here, the assumption is that the id of Kafka partitions are always
		// starting from 0, and therefore can be used directly as the offset
clockwise from the start index
		return (startIndex + partition.getPartition()) % numParallelSubtasks;

It seems it will assign to multi sub tasks.
I wonder how flink ensure some subtasks will simply remain idle

Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

View raw message