kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ma Tianchi (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (KAFKA-5319) Add a tool to make cluster replica and leader balance
Date Wed, 24 May 2017 11:15:04 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ma Tianchi reopened KAFKA-5319:

> Add a tool to make cluster replica and leader balance
> -----------------------------------------------------
>                 Key: KAFKA-5319
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5319
>             Project: Kafka
>          Issue Type: Improvement
>          Components: admin
>    Affects Versions:
>            Reporter: Ma Tianchi
>         Attachments: ClusterBalanceCommand.scala
> When a new broker is added to cluster,there is not any topics in the new broker.When
we use console command to create a topic without 'replicaAssignmentOpt',Kafka use AdminUtils.assignReplicasToBrokers
to get replicaAssignment.Even though it is balance at the creating time if the cluster never
change,with more and more brokers added to cluster the replica balanced will become not well.
We also can use 'kafka-reassign-partitions.sh' to balance ,but the big amount of topics make
it not easy.And at the topic created time , Kafka choose a  PreferredReplicaLeader which be
put at the first position of  the AR to make leader balance.But the balance will be destroyed
when cluster changed.Using  'kafka-reassign-partitions.sh' to make partition reassignment
may be also destroy the leader balance ,because user can change the AR of the partition .
It may be not balanced , but Kafka believe cluster leader balance is well with every leaders
is the first on at AR.
> So we create a tool to make the number of replicas and number of leaders on every brokers
is balanced.It uses a algorithm to get a balanced replica and leader  reassignment,then uses
ReassignPartitionsCommand to real balance the cluster.
> It can be used to make balance when cluster added brokers or cluster is not balanced
.It does not deal with moving replicas of a dead broker to an alive broker,it only makes replicas
and leaders on alive brokers is balanced.

This message was sent by Atlassian JIRA

View raw message