Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 431531817D for ; Thu, 28 Jan 2016 03:15:41 +0000 (UTC) Received: (qmail 97782 invoked by uid 500); 28 Jan 2016 03:15:40 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 97630 invoked by uid 500); 28 Jan 2016 03:15:40 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 97583 invoked by uid 99); 28 Jan 2016 03:15:40 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2016 03:15:40 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id DA4E02C1F54 for ; Thu, 28 Jan 2016 03:15:39 +0000 (UTC) Date: Thu, 28 Jan 2016 03:15:39 +0000 (UTC) From: "tangshangwen (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HDFS-9717) NameNode can not update the status of bad block MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 tangshangwen created HDFS-9717: ---------------------------------- Summary: NameNode can not update the status of bad block Key: HDFS-9717 URL: https://issues.apache.org/jira/browse/HDFS-9717 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: tangshangwen Assignee: tangshangwen In our cluster=EF=BC=8Csome users set the number of replicas of file to 1, = then back to 2=EF=BC=8Cthe file cannot be read=EF=BC=8Cbut the NameNode thi= nk it is healthy {noformat} /user/username/dt=3D2015-11-30/dp=3D16/part-r-00063.lzo 1513716944 bytes, 1= 2 block(s): Under replicated BP-1422437282658:blk_1897961957_824575827. Ta= rget Replicas is 2 but found 1 replica(s). Replica placement policy is violated for BP-1422437282658:blk_1897961957_8= 24575827. Block should be additionally replicated on 1 more rack (s). 0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694 len= =3D134217728 repl=3D2 [host1:50010, host2:50010] 1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 len= =3D134217728 repl=3D1 [host3:50010] 2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917 len= =3D134217728 repl=3D2 [host4:50010, host1:50010] ...... Status: HEALTHY Total size: 1513716944 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 12 (avg. block size 126143078 B) Minimally replicated blocks: 12 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 1 (8.333333 %) Mis-replicated blocks: 1 (8.333333 %) Default replication factor: 3 Average block replication: 1.9166666 Corrupt blocks: 0 Missing replicas: 1 (4.1666665 %) Number of data-nodes: xxxx Number of racks: xxx FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds {noformat} But the replica on the datanode has been damaged, can't read=EF=BC=8Cthis = is datanode log {noformat} 2016-01-23 06:34:42,737 WARN org.apache.hadoop.hdfs.server.datanode.BlockPo= olSliceScanner: First Verification failed for BP-1337805335-xxx.xxx.xxx.xxx= -1422437282658:blk_1897961957_824575827 = = =20 java.io.IOException: Input/output error = = =20 at java.io.FileInputStream.readBytes(Native Method) = = =20 at java.io.FileInputStream.read(FileInputStream.java:272) = = =20 at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) = = =20 at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockS= ender.java:529) = =20 at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSe= nder.java:710) = =20 at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyB= lock(BlockPoolSliceScanner.java:427) = =20 at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyF= irstBlock(BlockPoolSliceScanner.java:506) = =20 at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(Bl= ockPoolSliceScanner.java:667) = =20 at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlo= ckPoolSlice(BlockPoolSliceScanner.java:633) = =20 at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBloc= kScanner.java:101) = =20 at java.lang.Thread.run(Thread.java:745) ------------------------------ 2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNod= e: DatanodeRegistration(host1, storageID=3DDS-1450783279-xxx.xxx.xxx.xxx-50= 010-1432889625435 , infoPort=3D50075, ipcPort=3D50020, storageInfo=3Dlv=3D-47;cid=3DCID-3f363= 97d-b160-4414-b7e4-f37b72e96d53;nsid=3D1992344832;c=3D0):Failed to transfer= BP-1337805335-xxx.xxx.xxx.xxx-142243 7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got java.net.SocketException: Original Exception : java.io.IOException: Input/o= utput error at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:4= 33) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutpu= tStream.java:223) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockS= ender.java:546) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSe= nder.java:710) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(Dat= aNode.java:1520) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Input/output error {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)