kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3438) Rack Aware Replica Reassignment should warn of overloaded brokers
Date Sat, 23 Sep 2017 04:49:04 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Guozhang Wang updated KAFKA-3438:

*Reminder to the contributor / reviewer of the PR*: please note that the code deadline for
1.0.0 is less than 2 weeks away (Oct. 4th). Please re-evaluate your JIRA and see if it still
makes sense to be merged into 1.0.0 or it could be pushed out to 1.1.0, or be closed directly
if the JIRA itself is not valid any more, or re-assign yourself as contributor / committer
if you are no longer working on the JIRA.

> Rack Aware Replica Reassignment should warn of overloaded brokers
> -----------------------------------------------------------------
>                 Key: KAFKA-3438
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3438
>             Project: Kafka
>          Issue Type: Improvement
>    Affects Versions:
>            Reporter: Ben Stopford
>            Assignee: Vahid Hashemian
>             Fix For: 1.0.0
> We've changed the replica reassignment code to be rack aware.
> One problem that might catch users out would be that they rebalance the cluster using
kafka-reassign-partitions.sh but their rack configuration means that some high proportion
of replicas are pushed onto a single, or small number of, brokers. 
> This should be an easy problem to avoid, by changing the rack assignment information,
but we should probably warn users if they are going to create something that is unbalanced.

> So imagine I have a Kafka cluster of 12 nodes spread over two racks with rack awareness
enabled. If I add a 13th machine, on a new rack, and run the rebalance tool, that new machine
will get ~6x as many replicas as the least loaded broker. 
> Suggest a warning  be added to the tool output when --generate is called. "The most loaded
broker has 2.3x as many replicas as the the least loaded broker. This is likely due to an
uneven distribution of brokers across racks. You're advised to alter the rack config so there
are approximately the same number of brokers per rack" and displays the individual rack→#brokers
and broker→#replicas data for the proposed move.  

This message was sent by Atlassian JIRA

View raw message