Return-Path: 
 <hdfs-issues-return-144359-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9EE4318383
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Fri,  6 Nov 2015 19:14:11 +0000 (UTC)
Received: (qmail 42486 invoked by uid 500); 6 Nov 2015 19:14:11 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 42405 invoked by uid 500); 6 Nov 2015 19:14:11 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 42124 invoked by uid 99); 6 Nov 2015 19:14:11 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2015 19:14:11 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 0CCD42C1F5F
	for <hdfs-issues@hadoop.apache.org>; Fri,  6 Nov 2015 19:14:11 +0000 (UTC)
Date: Fri, 6 Nov 2015 19:14:11 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12904594.1444755450000.1716.1446837251049@Atlassian.JIRA>
In-Reply-To: <JIRA.12904594.1444755450000@Atlassian.JIRA>
References: <JIRA.12904594.1444755450000@Atlassian.JIRA>
 <JIRA.12904594.1444755450349@arcas>
Subject: [jira] [Updated] (HDFS-9236) Missing sanity check for block size
 during block recovery
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HDFS-9236?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yongjun Zhang updated HDFS-9236:
--------------------------------
    Target Version/s: 2.8.0  (was: 2.7.3)

> Missing sanity check for block size during block recovery
> ---------------------------------------------------------
>
>                 Key: HDFS-9236
>                 URL: https://issues.apache.org/jira/browse/HDFS-9236
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: HDFS
>    Affects Versions: 2.7.1
>            Reporter: Tony Wu
>            Assignee: Tony Wu
>         Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch, HDFS-9236.=
003.patch, HDFS-9236.004.patch, HDFS-9236.005.patch, HDFS-9236.006.patch, H=
DFS-9236.007.patch
>
>
> Ran into an issue while running test against faulty data-node code.=20
> Currently in DataNode.java:
> {code:java}
>   /** Block synchronization */
>   void syncBlock(RecoveringBlock rBlock,
>                          List<BlockRecord> syncList) throws IOException {
> =E2=80=A6
>     // Calculate the best available replica state.
>     ReplicaState bestState =3D ReplicaState.RWR;
> =E2=80=A6
>     // Calculate list of nodes that will participate in the recovery
>     // and the new block size
>     List<BlockRecord> participatingList =3D new ArrayList<BlockRecord>();
>     final ExtendedBlock newBlock =3D new ExtendedBlock(bpid, blockId,
>         -1, recoveryId);
>     switch(bestState) {
> =E2=80=A6
>     case RBW:
>     case RWR:
>       long minLength =3D Long.MAX_VALUE;
>       for(BlockRecord r : syncList) {
>         ReplicaState rState =3D r.rInfo.getOriginalReplicaState();
>         if(rState =3D=3D bestState) {
>           minLength =3D Math.min(minLength, r.rInfo.getNumBytes());
>           participatingList.add(r);
>         }
>       }
>       newBlock.setNumBytes(minLength);
>       break;
> =E2=80=A6
>     }
> =E2=80=A6
>     nn.commitBlockSynchronization(block,
>         newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, fals=
e,
>         datanodes, storages);
>   }
> {code}
> This code is called by the DN coordinating the block recovery. In the abo=
ve case, it is possible for none of the rState (reported by DNs with copies=
 of the replica being recovered) to match the bestState. This can either be=
 caused by faulty DN code or stale/modified/corrupted files on DN. When thi=
s happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See FSNames=
ystem.java:
> {code:java}
>   void commitBlockSynchronization(ExtendedBlock oldBlock,
>       long newgenerationstamp, long newlength,
>       boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
>       String[] newtargetstorages) throws IOException {
> =E2=80=A6
>       if (deleteblock) {
>         Block blockToDel =3D ExtendedBlock.getLocalBlock(oldBlock);
>         boolean remove =3D iFile.removeLastBlock(blockToDel) !=3D null;
>         if (remove) {
>           blockManager.removeBlock(storedBlock);
>         }
>       } else {
>         // update last block
>         if(!copyTruncate) {
>           storedBlock.setGenerationStamp(newgenerationstamp);
>          =20
>           //>>>> XXX block length is updated without any check <<<<//
>           storedBlock.setNumBytes(newlength);
>         }
> =E2=80=A6
>     if (closeFile) {
>       LOG.info("commitBlockSynchronization(oldBlock=3D" + oldBlock
>           + ", file=3D" + src
>           + (copyTruncate ? ", newBlock=3D" + truncatedBlock
>               : ", newgenerationstamp=3D" + newgenerationstamp)
>           + ", newlength=3D" + newlength
>           + ", newtargets=3D" + Arrays.asList(newtargets) + ") successful=
");
>     } else {
>       LOG.info("commitBlockSynchronization(" + oldBlock + ") successful")=
;
>     }
>   }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent =
block report (even with correct length) will cause the block to be marked a=
s corrupted. Since this is block could be the last block of the file. If th=
is happens and the client goes away, NN won=E2=80=99t be able to recover th=
e lease and close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN=
 to prevent such case from happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)