hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert J Berger <rber...@runa.com>
Subject Persistent small number of Blocks with corrupt replicas / Under replicated blocks
Date Wed, 08 Jun 2011 17:38:46 GMT
Synopsis: 
* After shutting down a datanode in  a cluster, fsck declares CORRUPT with missing blocks,

* I restore/restart the datanode and fsck soon declares things healthy
* But dfsadmin -report says a small number of blocks have corrupt replicas and an even smaller
number of under replicated blocks
* After a couple of days that number corrupt replicas and under replicated blocks stays the
same

Full Story:
My Goal is to rebalance blocks across 3 drives each within 2 datanodes in a 9 datanode (Replication=3)
cluster running hadoop 0.20.1
(EBS Volumes were added to the datanodes over time so one disk had 95% usage and the others
had significantly less)

The plan was to decommission the nodes and then wipe the disks and then add them back in to
the cluster.

Before I started I ran fsck and all was healthy. (Unfortunately I did not really look at the
dfsadmin -report at that time, so I can't be sure if there were no blocks with corrupt replicas
at this point)

I put two nodes into the Decommission process and after waiting about 36 hours it hadn't finished
decommissioning ether. So I decided to throw caution to the wind and shut down one of them.
(and had taken the node I was shutting down  out of the dfs.exclude.file file, also removed
the 2nd node from the dfs.exclude.file , dfsadmin -refreshNodes but kept the 2nd node live)

After shutting down one node, running fsck showed about 400 blocks as missing.

So I brought back up the shutdown node (it took a while as I had to restore it from EBS snapshot)
and fsck quickly went back to healthy but with a significant amount of Over replicated blocks

I put that node back into the decommissioning state (put just that one node back in the dfs.exclude.file
and ran dfsadmin -refreshNodes.

After another day or so, its still in the decommissioning mode. Fsck says the cluster is healthy
but still 37% over-replicated blocks. 

But the thing that concerns me is that  dfsadmin -report says:

Under replicated blocks: 18
Blocks with corrupt replicas: 34

So really two questions:

* Is there a way to force these corrupt replicas and under replicated blocks to get fixed?
* Is there a way to speed up the decommissioning process (without restarting the cluster)

I presume that its not safe for me to take down this node until the decommissioning completes
and/or the corrupt replicas are fixed..

And finally, is there a better way to accomplish the original task of rebalancing disks on
a datanode?

Thanks!
Rob
__________________
Robert J Berger - CTO
Runa Inc.
+1 408-838-8896
http://blog.ibd.com




Mime
View raw message