kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Kumar <varunkumar...@hotmail.com>
Subject Partition Reassignment in Cloud
Date Thu, 20 Jun 2019 03:50:30 GMT
Hi

I have been trying a small experiment with partition reassignment in cloud. where instead
of copying data between brokers using network, I moved the disk between the 2 brokers and
ran the partition reassignment. This actually increased the speed of partition reassignment
significantly. (As it had to catchup/fetch only down time data)


I tried this experiment this in Kafka 2.2.1 and it worked. I validated the data-consistency
using "kafka-replica-verification.sh" script.

Few more details of the experiment:

  *   Both the brokers from and to which the partitions are moving had to be shutdown.
  *   All the partitions in the disk are moved at once to new broker.
  *   Had to update broker.id property in meta.properties file for the moved log directory
before broker restart .
  *   Had to re-balance Leaders after brokers restart.

Can you please let me know if this approach will work in production ? Is there any scenario
where it might truncate/delete all data in moved disk and copy complete partition over network
?

Thanks
Varun



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message