Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 17 May 2017 20:31:04 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13071553.1494599392000.234465.1495053064277@Atlassian.JIRA>
In-Reply-To: <JIRA.13071553.1494599392000@Atlassian.JIRA>
References: <JIRA.13071553.1494599392000@Atlassian.JIRA> <JIRA.13071553.1494599392036@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-11817) A faulty node can cause a lease
 leak and NPE on accessing data
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 17 May 2017 20:31:09 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014712#comment-16014712 ] 

Kihwal Lee commented on HDFS-11817:
-----------------------------------

*Summary:*
 The observation of the incident resulted in discovery of three flaws.

1) A block recovery can involve dead nodes and it can lead to a corruption of a data structure, which causes NPE.
2) If a block cannot be completed, {{commitBlockSynchronization()}} will fail, since it requires all blocks to be complete, unlike regular file closings.
3) If a block has experienced 2) and remains committed (not complete), next lease recovery will result in a lease state corruption. (Removed from LeaseManager, but INode stays under-construction)


> A faulty node can cause a lease leak and NPE on accessing data
> --------------------------------------------------------------
>
>                 Key: HDFS-11817
>                 URL: https://issues.apache.org/jira/browse/HDFS-11817
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.8.0
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>
> When the namenode performs a lease recovery for a failed write, the {{commitBlockSynchronization()}} will fail, if none of the new target has sent a received-IBR.  At this point, the data is inaccessible, as the namenode will throw a {{NullPointerException}} upon {{getBlockLocations()}}.
> The lease recovery will be retried in about an hour by the namenode. If the nodes are faulty (usually when there is only one new target), they may not block report until this point. If this happens, lease recovery throws an {{AlreadyBeingCreatedException}}, which causes LeaseManager to simply remove the lease without  finalizing the inode.  
> This results in an inconsistent lease state. The inode stays under-construction, but no more lease recovery is attempted. A manual lease recovery is also not allowed. 


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org