kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4402) Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments "If no partition or key is present choose a partition in a round-robin fashion"
Date Mon, 02 Jan 2017 04:50:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15792204#comment-15792204
] 

ASF GitHub Bot commented on KAFKA-4402:
---------------------------------------

GitHub user yaojuncn reopened a pull request:

    https://github.com/apache/kafka/pull/2128

    KAFKA-4402: make the KafkaProducer true round robin per topic

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yaojuncn/kafka KAFKA-4402-client-producer-round-robin-fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/2128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2128
    
----
commit d74b0bd4d712223da38b3f2e666e4611bf65b455
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Date:   2016-11-13T21:46:44Z

    KAFKA-4402: make the KafkaProducer true round robin per topic

commit fdd946a0969ef9aed9f8e4bcb8ea095b81107411
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Date:   2016-11-13T22:18:09Z

    KAFKA-4402: fix some style issues

commit f978b50b58ff14ee41a0d7b360073bf4671c2bc2
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Date:   2016-11-14T05:26:00Z

    [KAFKA-4402] add performance test for round-robin default partitioner,
    the perf test result on my own laptop is: "avg partition time(ms)=0.000109", so it's pretty
fast with very low overhead that can be ignored,
    anyway, from code perspective it only added one ConcurrentHashMap.get(topic)

commit f9ca41152b6241ead91b663289198a58ef0454d5
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Date:   2016-11-19T05:38:37Z

    Merge branch 'trunk' into KAFKA-4402-client-producer-round-robin-fix

commit 82421e4b8281bc3236880a5296c3bd7c511e6c33
Author: yaojuncn <yaojuncn@users.noreply.github.com>
Date:   2017-01-01T20:38:20Z

    KAFKA-4402: fix some style issues in DefaultPartitionerTest.java

----


> Kafka Producer's DefaultPartitioner is actually not round robin as said in the code comments
"If no partition or key is present choose a partition in a round-robin fashion"
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-4402
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4402
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Jun Yao
>            Priority: Minor
>
> From this code comments, it is said that Kafka client  Producer's DefaultPartitioner
will do round robin if "no partition or key is present", 
> https://github.com/apache/kafka/blob/41e676d29587042994a72baa5000a8861a075c8c/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L34
> from the code it looks trying to do round robin as well, as it maintained a counter and
try to increase it every time and then will decide which partition to go to; 
> However the issue here is the counter is a global counter that is shared by all the topics,
so  it is actually not round robin per topic and sometimes caused unbalanced routing among
different partitions. 
> Although we can pass a custom implementation of interface "org.apache.kafka.clients.producer.Partitioner",
it might be still good to make the default implementation true round robin as comment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message