kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viktor Somogyi-Vass <viktorsomo...@gmail.com>
Subject [DISCUSS] KIP-435: Incremental Partition Reassignment
Date Fri, 22 Feb 2019 13:34:04 GMT
Hi Folks,

I've created a KIP about an improvement of the reassignment algorithm we
have. It aims to enable partition-wise incremental reassignment. The
motivation for this is to avoid excess load that the current replication
algorithm implicitly carries as in that case there are points in the
algorithm where both the new and old replica set could be online and
replicating which puts double (or almost double) pressure on the brokers
which could cause problems.
Instead my proposal would slice this up into several steps where each step
is calculated based on the final target replicas and the current replica
assignment taking into account scenarios where brokers could be offline and
when there are not enough replicas to fulfil the min.insync.replica
requirement.

The link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment

I'd be happy to receive any feedback.

An important note is that this KIP and another one, KIP-236 that is about
interruptible reassignment (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment)
should be compatible.

Thanks,
Viktor

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message