hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8224) Any IOException in DataTransfer#run() will run diskError thread even if it is not disk error
Date Wed, 22 Apr 2015 19:53:59 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rushabh S Shah updated HDFS-8224:
---------------------------------
    Description: 
This happened in our 2.6 cluster.
One of the block and its metadata file were corrupted.
Namenode tried to copy that block to another datanode but failed with the following stack
trace:

2015-04-20 01:04:04,421 [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4]
WARN datanode.DataNode: DatanodeRegistration(a.b.c.d, datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7,
infoPort=1006, infoSecurePort=0, ipcPort=8020, storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to a1.b1.c1.d1:1004 got 
java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0
        at org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:287)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
        at java.lang.Thread.run(Thread.java:722)

The following catch block in DataTransfer#run method will treat every IOException as disk
error fault and run disk errror
{noformat}
catch (IOException ie) {
        LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
            targets[0] + " got ", ie);
        // check if there are any disk problem
        checkDiskErrorAsync();
      } 
{noformat}
This block was never scanned by BlockPoolSliceScanner otherwise it would have reported as
corrupt block.









  was:
This happened in our 2.6 cluster.
One of the block and its metadata file were corrupted.
Namenode tried to copy that block to another datanode but failed with the following stack
trace:

2015-04-20 01:04:04,421 [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4]
WARN datanode.DataNode: DatanodeRegistration(a.b.c.d, datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7,
infoPort=1006, infoSecurePort=0, ipcPort=8020, storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to a1.b1.c1.d1:1004 got 
java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0
        at org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:287)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
        at java.lang.Thread.run(Thread.java:722)

The catch block in DataTransfer#run method will treat every IOException as disk error fault
and run disk errror

catch (IOException ie) {
        LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
            targets[0] + " got ", ie);
        // check if there are any disk problem
        checkDiskErrorAsync();
      } 

This block was never scanned by BlockPoolSliceScanner otherwise it would have reported as
corrupt block.










> Any IOException in DataTransfer#run() will run diskError thread even if it is not disk
error
> --------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8224
>                 URL: https://issues.apache.org/jira/browse/HDFS-8224
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Rushabh S Shah
>             Fix For: 2.8.0
>
>
> This happened in our 2.6 cluster.
> One of the block and its metadata file were corrupted.
> Namenode tried to copy that block to another datanode but failed with the following stack
trace:
> 2015-04-20 01:04:04,421 [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4]
WARN datanode.DataNode: DatanodeRegistration(a.b.c.d, datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7,
infoPort=1006, infoSecurePort=0, ipcPort=8020, storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to a1.b1.c1.d1:1004 got 
> java.io.IOException: Could not create DataChecksum of type 0 with bytesPerChecksum 0
>         at org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
>         at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
>         at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
>         at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
>         at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:287)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
>         at java.lang.Thread.run(Thread.java:722)
> The following catch block in DataTransfer#run method will treat every IOException as
disk error fault and run disk errror
> {noformat}
> catch (IOException ie) {
>         LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
>             targets[0] + " got ", ie);
>         // check if there are any disk problem
>         checkDiskErrorAsync();
>       } 
> {noformat}
> This block was never scanned by BlockPoolSliceScanner otherwise it would have reported
as corrupt block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message