kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nehanarkh...@apache.org
Subject svn commit: r1577423 - /kafka/site/081/ops.html
Date Fri, 14 Mar 2014 04:33:37 GMT
Author: nehanarkhede
Date: Fri Mar 14 04:33:37 2014
New Revision: 1577423

URL: http://svn.apache.org/r1577423
Log:
Updated docs for the partition reassignment tool

Modified:
    kafka/site/081/ops.html

Modified: kafka/site/081/ops.html
URL: http://svn.apache.org/viewvc/kafka/site/081/ops.html?rev=1577423&r1=1577422&r2=1577423&view=diff
==============================================================================
--- kafka/site/081/ops.html (original)
+++ kafka/site/081/ops.html Fri Mar 14 04:33:37 2014
@@ -117,15 +117,82 @@ Adding servers to a Kafka cluster is eas
 The process of migrating data is manually initiated but fully automated. Under the covers
what happens is that Kafka will add the new server as a follower of the partition it is migrating
and allow it to fully replicate the existing data in that partition. When the new server has
fully replicated the contents of this partition and joined the in-sync replica one of the
existing replicas will delete their partition's data.
 
 <h5>Automatically migrating data to new machines</h5>
-Cover the case where we just want to migrate a fair share to the new machines.
+The partition reassignment tool can be used to move some topics off of the current set of
brokers to the newly added brokers. When used to do this, the user should provide a list of
topics that should be moved to the new set of brokers and a target list of new brokers. The
tool then evenly distributes all partitions for the given list of topics to the new set of
brokers. During this move, the replication factor of the topic is kept constant. Effectively
the replicas for all partitions are moved from the old set of brokers to the newly added brokers.

 
-Neha will write this :-)
+For example, the following will move all partitions for topics foo1,foo2 to the new set of
brokers 5,6. At the end of this move, all partitions for topics foo1 and foo2 will only exist
on brokers 5,6
+<pre>
+bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --broker-list
"5,6" --generate 
+Current partition replica assignment
+
+{"version":1,"partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},{"topic":"foo1","partition":0,"replicas":[3,4]},{"topic":"foo2","partition":2,"replicas":[1,2]},{"topic":"foo2","partition":0,"replicas":[3,4]},{"topic":"foo1","partition":1,"replicas":[2,3]},{topic":"foo2","partition":1,"replicas":[2,3]}]}
+
+Proposed partition reassignment configuration
+
+{"version":1,"partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":2,"replicas":[5,6]},{"topic":"foo2","partition":0,"replicas":[5,6]},{"topic":"foo1","partition":1,"replicas":[5,6]},{topic":"foo2","partition":1,"replicas":[5,6]}]}
+
+cat topics-to-move.json
+{"topics":
+     [{"topic": "foo1"},{"topic": "foo2"}],
+     "version":1
+}
+</pre>
+<p>
+The tool generates a candidate assignment that will move all partitions from topics foo1,foo2
to brokers 5,6. Note, however, that at this point, the partition movement has not started,
it merely tells you the current assignment and the proposed new assignment. The current assignment
should be saved in case you want to rollback to it. The new assignment should be input to
the tool with the --execute option as follows
+
+<pre>
+bin/kafka-reassign-partitions.sh --reassignment-json-file expand-cluster-reassignment.json
--execute
+Current partition replica assignment
+
+{"version":1,"partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},{"topic":"foo1","partition":0,"replicas":[3,4]},{"topic":"foo2","partition":2,"replicas":[1,2]},{"topic":"foo2","partition":0,"replicas":[3,4]},{"topic":"foo1","partition":1,"replicas":[2,3]},{topic":"foo2","partition":1,"replicas":[2,3]}]}
+
+Save this to use as the --reassignment-json-file option during rollback
+Successfully started reassignment of partitions
+{"version":1,"partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":2,"replicas":[5,6]},{"topic":"foo2","partition":0,"replicas":[5,6]},{"topic":"foo1","partition":1,"replicas":[5,6]},{topic":"foo2","partition":1,"replicas":[5,6]}]}
+</pre>
+<p>
+The --verify option can be used with the tool to check the status of the partition reassignment.
Note that the same expand-cluster-reassignment.json (used with the --execute option) should
be used with the --verify option
+
+<pre>
+bin/kafka-reassign-partitions.sh --reassignment-json-file expand-cluster-reassignment.json
--verify
+Status of partition reassignment:
+Reassignment of partition [foo1,0] completed successfully
+Reassignment of partition [foo1,1] is in progress
+Reassignment of partition [foo1,2] is in progress
+Reassignment of partition [foo2,0] completed successfully
+Reassignment of partition [foo2,1] completed successfully 
+Reassignment of partition [foo2,2] completed successfully 
+</pre>
 
 <h5>Custom partition assignment and migration</h5>
+The partition reassignment tool can also be used to selectively move replicas of a partition
to a specific set of brokers. When used in this manner, it is assumed that the user knows
the reassignment and does not require the tool to generate a candidate reassignment, effectively
skipping the --generate step and moving straight to the --execute step
+<p>
+For example, the following moves partition 0 of topic foo1 to brokers 5,6 and partition 1
of topic foo2 to brokers 2,3
+
+<pre>
+cat custom-reassignment.json
+{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}
+
+bin/kafka-reassign-partitions.sh --reassignment-json-file custom-reassignment.json --execute
+Current partition replica assignment
 
-Neha will write this :-)
+{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},{"topic":"foo2","partition":1,"replicas":[3,4]}]}
+
+Save this to use as the --reassignment-json-file option during rollback
+Successfully started reassignment of partitions
+{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}
+</pre>
+<p>
+The --verify option can be used with the tool to check the status of the partition reassignment.
Note that the same expand-cluster-reassignment.json (used with the --execute option) should
be used with the --verify option
+
+<pre>
+bin/kafka-reassign-partitions.sh --reassignment-json-file custom-reassignment.json --verify
+Status of partition reassignment:
+Reassignment of partition [foo1,0] completed successfully
+Reassignment of partition [foo2,1] completed successfully 
+</pre>
 
-Does this cover decommissioning machines? If so we should mention that...
+<h5>Decommissioning machines</h5>
+The partition reassignment tool does not have the ability to decommission machines yet. As
such, you cannot decommission machines without effectively reducing the replication factor
of the partitions that existed on the decommissioned machine. We plan to add support for this
in 0.8.2
 
 <h3><a id="datacenters">6.2 Datacenters</a></h3>
 



Mime
View raw message