flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Use a round-robin kafka partitioner
Date Wed, 25 Oct 2017 07:14:50 GMT
Hi!

you will have to modify your partitioner to implement the 
FlinkKafkaPartitioner 
<https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/partitioner/FlinkKafkaPartitioner.java>interface

instead.
You can then plug this into any kafka sink through on of the constructors.

Regards,
Chesnay

On 24.10.2017 22:15, kla wrote:
> Hi Chesnay,
>
> Thanks for your reply.
>
> I would like to use the partitioner within the Kafka Sink operation.
>
> By default kafka sink is using FixedPartitioner:
>
> 	public FlinkKafkaProducer010(String topicId, KeyedSerializationSchema<T>
> serializationSchema, Properties producerConfig) {
> 		this(topicId, serializationSchema, producerConfig, new
> FixedPartitioner<T>());
> 	}
>
> So I have 12 kafka topic partitions and I have 2 Flink partitions, and I
> have unbalanced partitioning.
> According to the java doc in the FixedPartitioner class which is following:
>
>   *  Not all Kafka partitions contain data
>   *  To avoid such an unbalanced partitioning, use a round-robin kafka
> partitioner. (note that this will
>   *  cause a lot of network connections between all the Flink instances and
> all the Kafka brokers
>
> According to the this I have to use a round-robin kafka partitioner. And
> what is the right way to do it ?
>
> Thanks again.
>
>
>
> --
> Sent from:http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>


Mime
View raw message