kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <ka...@harsha.io>
Subject Re: [DISCUSS] KIP-435: Incremental Partition Reassignment
Date Fri, 22 Feb 2019 16:05:38 GMT
Hi Viktor,
            There is already KIP-236 for the same feature and George made a PR for this as
Lets consolidate these two discussions. If you have any cases that are not being solved by
KIP-236 can you please mention them in that thread. We can address as part of KIP-236.


On Fri, Feb 22, 2019, at 5:44 AM, Viktor Somogyi-Vass wrote:
> Hi Folks,
> I've created a KIP about an improvement of the reassignment algorithm we
> have. It aims to enable partition-wise incremental reassignment. The
> motivation for this is to avoid excess load that the current replication
> algorithm implicitly carries as in that case there are points in the
> algorithm where both the new and old replica set could be online and
> replicating which puts double (or almost double) pressure on the brokers
> which could cause problems.
> Instead my proposal would slice this up into several steps where each step
> is calculated based on the final target replicas and the current replica
> assignment taking into account scenarios where brokers could be offline and
> when there are not enough replicas to fulfil the min.insync.replica
> requirement.
> The link to the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment
> I'd be happy to receive any feedback.
> An important note is that this KIP and another one, KIP-236 that is 
> about
> interruptible reassignment (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment)
> should be compatible.
> Thanks,
> Viktor

View raw message