hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: "Could not obtain block" error
Date Wed, 29 Oct 2008 18:46:27 GMT

If have only one copy of the block and it is mostly corrupted.. Namenode 
itself can not correct it. Of course, DFSClient should not print error 
in a infinite loop.

I think there was an old bug where crc file got overwritten by 0 length 
file.

One work around for you is to go to the datanode and remove the .crc 
file for this block (find /datanodedir -name blk_5994030096182059653\*). 
Be careful not to remove the block file itself.

longer term fix : upgrade to more recent version.

Raghu.

murali krishna wrote:
> Hi,
> When I try to read one of the file from dfs, I get the following error in an infinite
loop (using 0.15.3)
> 
> “08/10/28 23:43:15 INFO fs.DFSClient: Could not obtain block blk_5994030096182059653
from any node:  java.io.IOException: No live nodes contain current block”
> 
> Fsck showed that the file is HEALTHY but under replicated (1 instead of configured 2).
I checked the datanode log where the only replica exists for that block and I can see repeated
errors while serving that bock.
> 2008-10-22 23:55:39,378 WARN org.apache.hadoop.dfs.DataNode: Failed to transfer blk_59940300961820596
> 53 to 68.142.212.228:50010 got java.io.EOFException
>         at java.io.DataInputStream.readShort(DataInputStream.java:298)
>         at org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1061)
>         at org.apache.hadoop.dfs.DataNode$DataTransfer.run(DataNode.java:1446)
>         at java.lang.Thread.run(Thread.java:619)
> 
> Any idea what is going on and how can I fix this ?
> 
> Thanks,
> Murali
> 


Mime
View raw message