Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Mon, 13 Oct 2014 23:08:33 +0000 (UTC)
From: "Ming Ma (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12746626.1412748975000.258426.1413241713968@Atlassian.JIRA>
In-Reply-To: <JIRA.12746626.1412748975000@Atlassian.JIRA>
References: <JIRA.12746626.1412748975000@Atlassian.JIRA>
 <JIRA.12746626.1412748975497@arcas>
Subject: [jira] [Updated] (HDFS-7208) NN doesn't schedule replication when a
 DN storage fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ming Ma updated HDFS-7208:
--------------------------
    Attachment: HDFS-7208.patch

Here is the initial patch based on heartbeat notification approach, the assumption is DN will report all healthy storages in the heartbeat. This approach is simpler than the blockReport approach which needs to have DN persist the info to cover some failure scenarios. It also makes storage failure detection faster.

1. NN detects failed storages during HB processing based on the delta between DN's reported healthy storages and the storages NN has. Marked the state of those missing storages DatanodeStorage.State.FAILED.

2. HeartbeatManager will remove blocks on those DatanodeStorage.State.FAILED storages. This will cover some corner scenarios where new replicas might be added to BlocksMap afterwards.

3. It also covers the case where admins reduce the number of healthy volumes on DN and restart DN.

> NN doesn't schedule replication when a DN storage fails
> -------------------------------------------------------
>
>                 Key: HDFS-7208
>                 URL: https://issues.apache.org/jira/browse/HDFS-7208
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ming Ma
>         Attachments: HDFS-7208.patch
>
>
> We found the following problem. When a storage device on a DN fails, NN continues to believe replicas of those blocks on that storage are valid and doesn't schedule replication.
> A DN has 12 storage disks. So there is one blockReport for each storage. When a disk fails, # of blockReport from that DN is reduced from 12 to 11. Given dfs.datanode.failed.volumes.tolerated is configured to be > 0, NN still considers that DN healthy.
> 1. A disk failed. All blocks of that disk are removed from DN dataset.
>  
> {noformat}
> 2014-10-04 02:11:12,626 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing replica BP-1748500278-xx.xx.xx.xxx-1377803467793:1121568886 on failed volume /data/disk6/dfs/current
> {noformat}
> 2. NN receives DatanodeProtocol.DISK_ERROR. But that isn't enough to have NN remove the DN and the replicas from the BlocksMap. In addition, blockReport doesn't provide the diff given that is done per storage.
> {noformat}
> 2014-10-04 02:11:12,681 WARN org.apache.hadoop.hdfs.server.namenode.NameNode: Disk error on DatanodeRegistration(xx.xx.xx.xxx, datanodeUuid=f3b8a30b-e715-40d6-8348-3c766f9ba9ab, infoPort=50075, ipcPort=50020, storageInfo=lv=-55;cid=CID-e3c38355-fde5-4e3a-b7ce-edacebdfa7a1;nsid=420527250;c=1410283484939): DataNode failed volumes:/data/disk6/dfs/current
> {noformat}
> 3. Run fsck on the file and confirm the NN's BlocksMap still has that replica.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)