kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-1829) Support adding new partitions on brokers without reassignment of existing partitions
Date Wed, 24 Dec 2014 23:02:13 GMT
Allen Wang created KAFKA-1829:
---------------------------------

             Summary: Support adding new partitions on brokers without reassignment of existing
partitions
                 Key: KAFKA-1829
                 URL: https://issues.apache.org/jira/browse/KAFKA-1829
             Project: Kafka
          Issue Type: New Feature
    Affects Versions: 0.8.1.1
            Reporter: Allen Wang


One solution to deal with increasing incoming traffic to Kafka brokers is to add new broker
instances and new partitions at the same time. This is possible if disk size is not the bottleneck.
In this case, ideally the new partitions would be assigned to new brokers without any changes
to existing partitions. The reason behind it is that moving existing partitions would cause
more traffic and hence more load.

The way to handle this now is a multi step process:

- add partitions
- generate replica assignment for the new partitions on the new brokers
- execute the assignment

This is complicated. Also, adding partitions and assigning replicas is not atomic -- when
partitions are added, they are already assigned using a default assignment algorithm.

This feature would greatly help auto scaling Kafka to deal with increasing load.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message