hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: hdfs question when replacing dead node...
Date Fri, 24 Jul 2009 02:21:12 GMT
How fast did you re-run fsck after re-joining the node? fsck returns data
based on the latest block reports from datanodes -- these are scheduled to
run (I think) every 15 minutes, so the NameNode's state on block replication
may be as much as 15 minutes out of date.

- Aaron

On Thu, Jul 23, 2009 at 3:00 PM, Andy Sautins
<andy.sautins@returnpath.net>wrote:

>
>   I recently had to replace a node on a hadoop 0.20.0 4-node cluster and I
> can't quite explain what happened.  If anyone has any insight I'd appreciate
> it.
>
>   When the node failed ( drive failure ) running the command 'hadoop fsck
> /' correctly showed the data nodes to now be 3 instead of 4 and showed the
> under replicated blocks to be replicated.  I assume that once the node was
> determined to be dead the blocks on the dead node were not considered in the
> replication factor and caused hdfs to replicate to the available nodes to
> meet the configured replication factor of 3.  All is good.  What I couldn't
> explain is that after re-building and re-starting the failed node I started
> the balancer ( bin/start-balancer.sh ) and re-ran 'hadoop fsck /'.  The
> number of nodes showed that the 4th node was now back in the cluster.  What
> struck me as strange is a large number of blocks ( > 2k ) were shown as
> under replicated.  The under replicated blocks were eventually re-replicated
> and all the data seems correct.
>
>   Can someone explain why re-adding a node that had died why the
> replication factor would go from 3 to 2?  Is there something with the
> balancer.sh script that would show fsck that the blocks are under
> replicated?
>
>   Note that I'm still getting the process for replacing failed nodes down
> so it's possible that I was looking at things wrong for a bit.
>
>    Any insight would be greatly appreciated.
>
>    Thanks
>
>    Andy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message