hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Wyss <wyss...@gmail.com>
Subject Re: How to fix Under-replication
Date Thu, 18 Apr 2013 00:45:26 GMT
In case anyone finds this tucked away on the internet in the future and is
in a situation like us....

We only had 3 racks, and 4 machines on one rack, compared with almost 70
nodes in our virtually provisioned service.

I found that hadoop calculates the maximum number of blocks per rack by

 int maxNodesPerRack =

The division is integer division. So with three racks, this amounts to
two blocks per rack. With 2 racks, you can store 3 blocks on each
We are decommissioning the 4 nodes so that even if we don't have rack
awareness, we'll at least get three copies.

The proper way to fix this is to adjust hadoop's topology script as
you can read about here:

Alternatively, HDFS-385 might be included in your distribution,
allowing you to control the block allocation policy.


On Wed, Apr 17, 2013 at 1:27 PM, Keith Wyss <wyssman@gmail.com> wrote:

> Hello there.
> I am operating a cluster that is consistently unable to create three
> replicas for a large percentage of blocks.
> I think I have a good idea of why this is the case, but I would like
> suggestions about how to fix it.
> First of all, we can begin with the namenode logs.
> There are lots of incidences of this statement:
> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to
> place enough replicas, still in need of 1
> The cluster is only just over 50% full and has well over 3 nodes. This and
> the lack of other widespread areas rules out the possibility that there is
> simply not room for the blocks.
> This leaves the possibility that the namenode is unable to satisfy the the
> block placement policy. I believe that this is what is happening.
> I read in
> http://www.slideshare.net/cloudera/hadoop-troubleshooting-101-kate-ting-cloudera
> that if there are more than 2 racks, then a block must be present on at
> least two racks.
> This makes sense, but our network situation is a little bizarre. It
> consists of:
> -a small amount of machines that have a dedicated datacenter/rack/host
> configuration
> -- These are spread across a few racks.
> -a large amount of machines that are provisioned using an internal
> hardware as a service provider.
> -- These are listed as one rack.
> The details of the rack allocation for the machines that are provisioned
> from the hardware as a service provider are abstracted away and are not
> attainable. The connection to the hardware provisioned as a service has a
> lot of bandwidth, so this is not as crazy as it sounds.
> Our problem is that the machines on all the smaller racks have now filled
> up the amount of space partitioned
> by dfs.datanode.du.reserved. This means that all of the blocks since these
> machines ran out of space are lacking one replication.
> Is there a way to configure Hadoop to create a third replication anyway
> (aside from changing the hadoop.topology.script implementation)?
> What can I do to either confirm or deny my suspicions?
> Thanks for your help,
> Keith

View raw message