hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <shawn.higg...@thomsonreuters.com>
Subject RE: How to find file with bad block during Distcp
Date Mon, 07 Nov 2011 21:59:14 GMT
You should be able to perform the following command:

grep "blk_4722582869815671042" ./* > blockFileName.txt

from your namenode log directory as the block ID's will be recorded in your namenode logs.
Then, look through blockFileName.txt to find the filename. It should be at the top if memory
serves me correctly.


From: Ayon Sinha [mailto:ayonsinha@yahoo.com]
Sent: Monday, November 07, 2011 3:04 PM
To: HDFS Mailinglist
Subject: How to find file with bad block during Distcp

How do I know which file has this block that is causing a Distcp to fail?

Copy failed: java.io.IOException: Fail to get block MD5 for blk_4722582869815671042_13395
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:844)
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:719)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:553)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:53)
        at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1261)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1120)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)


View raw message