kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Widman (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-2435) More optimally balanced partition assignment strategy
Date Thu, 08 Feb 2018 16:52:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357203#comment-16357203
] 

Jeff Widman edited comment on KAFKA-2435 at 2/8/18 4:51 PM:
------------------------------------------------------------

Can this be closed as wontfix? 

Two reasons:

 # it targets the deprecated (removed?) high-level consumer

# KIP-54 addressed some of the concerns here, and IMHO is a better solution because it addresses
both fairness and affinity/stickyness


was (Author: jeffwidman):
Can this be closed as wontfix? 

Two reasons:

 

1) it targets the deprecated (removed?) high-level consumer

2) KIP-54 addressed some of the concerns here, and IMHO is a better solution because it addresses
both fairness and affinity/stickyness

> More optimally balanced partition assignment strategy
> -----------------------------------------------------
>
>                 Key: KAFKA-2435
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2435
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Andrew Olson
>            Assignee: Andrew Olson
>            Priority: Major
>             Fix For: 2.0.0
>
>         Attachments: KAFKA-2435.patch
>
>
> While the roundrobin partition assignment strategy is an improvement over the range strategy,
when the consumer topic subscriptions are not identical (previously disallowed but will be
possible as of KAFKA-2172) it can produce heavily skewed assignments. As suggested [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
it would be nice to have a strategy that attempts to assign an equal number of partitions
to each consumer in a group, regardless of how similar their individual topic subscriptions
are. We can accomplish this by tracking the number of partitions assigned to each consumer,
and having the partition assignment loop assign each partition to a consumer interested in
that topic with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the partition assignment
order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions is assigned
first.
> The general idea behind these two rules is to keep the most flexible assignment choices
available as long as possible by starting with the most constrained partitions/consumers.
> This JIRA addresses the original high-level consumer. For the new consumer, see KAFKA-3297.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message