hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tangshangwen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9717) NameNode can not update the status of bad block
Date Thu, 28 Jan 2016 03:15:39 GMT
tangshangwen created HDFS-9717:
----------------------------------

             Summary: NameNode can not update the status of bad block
                 Key: HDFS-9717
                 URL: https://issues.apache.org/jira/browse/HDFS-9717
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.2.0
            Reporter: tangshangwen
            Assignee: tangshangwen


In our cluster,some users set the number of replicas of file to 1, then back to 2,the
file cannot be read,but the NameNode think it is healthy
{noformat}
/user/username/dt=2015-11-30/dp=16/part-r-00063.lzo 1513716944 bytes, 12 block(s):  Under
replicated BP-1422437282658:blk_1897961957_824575827. Target Replicas is 2 but found 1 replica(s).
 Replica placement policy is violated for BP-1422437282658:blk_1897961957_824575827. Block
should be additionally replicated on 1 more rack
(s).
0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694 len=134217728 repl=2
[host1:50010, host2:50010]
1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 len=134217728 repl=1
[host3:50010]
2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917 len=134217728 repl=2
[host4:50010, host1:50010]
......

Status: HEALTHY
 Total size:   1513716944 B
 Total dirs:   0
 Total files:  1
 Total symlinks:               0
 Total blocks (validated):     12 (avg. block size 126143078 B)
 Minimally replicated blocks:  12 (100.0 %)
 Over-replicated blocks:       0 (0.0 %)
 Under-replicated blocks:      1 (8.333333 %)
 Mis-replicated blocks:                1 (8.333333 %)
 Default replication factor:   3
 Average block replication:    1.9166666
 Corrupt blocks:               0
 Missing replicas:             1 (4.1666665 %)
 Number of data-nodes:         xxxx
 Number of racks:              xxx
FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds
{noformat}

But the  replica on the datanode has been damaged, can't read,this is datanode log
{noformat}
2016-01-23 06:34:42,737 WARN org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
First Verification failed for BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827
                                                                                         
                                                                
java.io.IOException: Input/output error                                                  
                                                                                 
    at java.io.FileInputStream.readBytes(Native Method)                                  
                                                                                 
    at java.io.FileInputStream.read(FileInputStream.java:272)                            
                                                                                 
    at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)                          
                                                                                 
    at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:529)
                                                                                
    at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
                                                                                 
    at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:427)
                                                           
    at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:506)
                                                      
    at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:667)
                                                                  
    at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:633)
                                                    
    at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
                                                                             
    at java.lang.Thread.run(Thread.java:745)
------------------------------
2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(host1,
storageID=DS-1450783279-xxx.xxx.xxx.xxx-50010-1432889625435
, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-3f36397d-b160-4414-b7e4-f37b72e96d53;nsid=1992344832;c=0):Failed
to transfer BP-1337805335-xxx.xxx.xxx.xxx-142243
7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got
java.net.SocketException: Original Exception : java.io.IOException: Input/output error
    at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
    at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)
    at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)
    at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
    at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546)
    at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710)
    at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1520)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Input/output error
{noformat}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message