hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Altering logical rack configuration
Date Tue, 08 Feb 2011 18:47:57 GMT

On Feb 4, 2011, at 5:00 PM, Jonathan Disher wrote:

> In my existing 48-node cluster, the engineer who originally designed it (no longer here)
did not specify logical racks in the HDFS configuration, instead leaving everything in "default-rack".
 Now I have 4 physical racks of machines, and I am becoming concerned about failure and near/far
replication issues.
> Anyone have any ideas what will happen if I tell hadoop about the physical rack layout
(i.e. nuke default-rack, and create rack110, rack111, rack112, rack113, etc)?

	What will happen is that your next fsck run will be full of replication policy violation
errors/warnings (I forget which).  These are non-fatal and trivial to fix if you have enough

a) use setrep to increase the replication factor of all files
b) let the namenode re-replicate all blocks
c) use setrep again to decrease the replication factor of all files
d) let the namenode remove an extra replica:  it will almost always choose a replica that
violates policy

	Is it worth fixing? Yes, definitely, for the reasons you suspect:  if you lose a rack, you
might lose data.
View raw message