hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: balancer and under replication
Date Mon, 07 Jan 2013 21:16:41 GMT
Under normal operation, NN takes care of under-replicated blocks by itself.

A file with a replication factor set higher than the cluster's nodes
will also register its blocks as under-replicated. A common config
mistake here is the mapred.submit.replication, which is a default of
10 (useful for 100 nodes but not otherwise), and you can verify via
fsck if these affected files are all staging directory MR files which
fall under this category. If so, just lowering their setrep will help.

If not the above, there's a chance that a rack misconfig may have
caused a bad state of replication (a violation of policy), which can
be fixed by the raise and subsequent lowering of the replication
factor as you state.

On Tue, Jan 8, 2013 at 2:18 AM, Patai Sangbutsarakum
<silvianhadoop@gmail.com> wrote:
> Hello Hadoopers,
> Currently my production cluster which is running cdh3u4 has shown
> Number of Under-Replicated Blocks around 1k blocks.
> Even though we have balancer run every night somehow the number of
> under replicate is never go down at all.
> The question is how HDFS handles under-replication blocks.
> - will namenode takes care when file that has under-replicated blocks
> is being used ?
> or
> - we need to bump up setrep to kind of trigger the number of replication block ?
> or
> - ??
> Thanks
> -P

Harsh J

View raw message