kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ij...@apache.org
Subject kafka git commit: MINOR: Documentation for Rack Awareness
Date Wed, 11 May 2016 14:31:33 GMT
Repository: kafka
Updated Branches:
  refs/heads/0.10.0 5bfba1dec -> 3178ecf11


MINOR: Documentation for Rack Awareness

Author: Ben Stopford <benstopford@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1369 from benstopford/rack-awareness-docs

(cherry picked from commit 6978115514fc74021e4f2c761402d6b0c954d50c)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/3178ecf1
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/3178ecf1
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/3178ecf1

Branch: refs/heads/0.10.0
Commit: 3178ecf11331e9e3e643c7ccdbd28afaf4b0e9b9
Parents: 5bfba1d
Author: Ben Stopford <benstopford@gmail.com>
Authored: Wed May 11 15:28:18 2016 +0100
Committer: Ismael Juma <ismael@juma.me.uk>
Committed: Wed May 11 15:31:26 2016 +0100

----------------------------------------------------------------------
 docs/ops.html | 11 +++++++++++
 1 file changed, 11 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka/blob/3178ecf1/docs/ops.html
----------------------------------------------------------------------
diff --git a/docs/ops.html b/docs/ops.html
index f64a701..faf5453 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -98,6 +98,17 @@ Since running this command can be tedious you can also configure Kafka
to do thi
     auto.leader.rebalance.enable=true
 </pre>
 
+<h4><a id="basic_ops_racks" href="#basic_ops_racks">Balancing Replicas Across
Racks</a></h4>
+The rack awareness feature spreads replicas of the same partition across different racks.
This extends the guarantees Kafka provides for broker-failure to cover rack-failure, limiting
the risk of data loss should all the brokers on a rack fail at once. The feature can also
be applied to other broker groupings such as availability zones in EC2.
+<p></p>
+You can specify that a broker belongs to a particular rack by adding a property to the broker
config:
+<pre>   broker.rack=my-rack-id</pre>
+When a topic is <a href="#basic_ops_add_topic">created</a>, <a href="#basic_ops_modify_topic">modified</a>
or replicas are <a href="#basic_ops_cluster_expansion">redistributed</a>, the
rack constraint will be honoured, ensuring replicas span as many racks as they can (a partition
will span min(#racks, replication-factor) different racks).
+<p></p>
+The algorithm used to assign replicas to brokers ensures that the number of leaders per broker
will be constant, regardless of how brokers are distributed across racks. This ensures balanced
throughput.
+<p></p>
+However if racks are assigned different numbers of brokers, the assignment of replicas will
not be even. Racks with fewer brokers will get more replicas, meaning they will use more storage
and put more resources into replication. Hence it is sensible to configure an equal number
of brokers per rack.
+
 <h4><a id="basic_ops_mirror_maker" href="#basic_ops_mirror_maker">Mirroring data
between clusters</a></h4>
 
 We refer to the process of replicating data <i>between</i> Kafka clusters "mirroring"
to avoid confusion with the replication that happens amongst the nodes in a single cluster.
Kafka comes with a tool for mirroring data between Kafka clusters. The tool reads from a source
cluster and writes to a destination cluster, like this:


Mime
View raw message