Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 2 May 2014 13:39:22 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12710666.1398461540473.219620.1399037962316@arcas>
In-Reply-To: <JIRA.12710666.1398461540473@arcas>
References: <JIRA.12710666.1398461540473@arcas>
Subject: [jira] [Commented] (HDFS-6289) HA failover can fail if there are
 pending DN messages for DNs which no longer exist
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987689#comment-13987689 ] 

Hudson commented on HDFS-6289:
------------------------------

FAILURE: Integrated in Hadoop-Hdfs-trunk #1748 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1748/])
HDFS-6289. HA failover can fail if there are pending DN messages for DNs which no longer exist. Contributed by Aaron T. Myers. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1591413)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingDataNodeMessages.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java


> HA failover can fail if there are pending DN messages for DNs which no longer exist
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-6289
>                 URL: https://issues.apache.org/jira/browse/HDFS-6289
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha
>    Affects Versions: 2.4.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Critical
>             Fix For: 2.5.0
>
>         Attachments: HDFS-6289.patch, HDFS-6289.patch
>
>
> In an HA setup, the standby NN may receive messages from DNs for blocks which the standby NN is not yet aware of. It queues up these messages and replays them when it next reads from the edit log or fails over. On a failover, all of these pending DN messages must be processed successfully in order for the failover to succeed. If one of these pending DN messages refers to a DN storageId that no longer exists (because the DN with that transfer address has been reformatted and has re-registered with the same transfer address) then on transition to active the NN will not be able to process this DN message and will suicide with an error like the following:
> {noformat}
> 2014-04-25 14:23:17,922 FATAL namenode.NameNode (NameNode.java:doImmediateShutdown(1525)) - Error encountered requiring NN shutdown. Shutting down immediately.
> java.io.IOException: Cannot mark blk_1073741825_900(stored=blk_1073741825_1001) as corrupt because datanode 127.0.0.1:33324 does not exist
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.2#6252)