hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
Date Thu, 15 Sep 2016 06:42:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492546#comment-15492546
] 

Konstantin Shvachko commented on HDFS-10301:
--------------------------------------------

Still not clear what scenario concerns you. Arpit, could you please clarify.
* Balancer copies a replica from a source DN to a target DN and when finished sends IBR with
the target as a new replica location and a hint to remove old replica from the source DN.
If the source or the target storage fails during this the transfer fails and Balancer moves
on. If either of the storages fail after the transfer it is the same as the regular failure,
the block will become under-replicated and recovered in due time.
* For VolumeChoosingPolicy it is even more important to know early which storages failed in
order to avoid choosing them as targets.

In fact the code path of zombie storage removal via FBRs (introduced by HDFS-7960) is practically
never triggered. Because heartbeats are much more often, the removal of zombies goes through
heartbeats. So if this is unsafe as you assume we should have the evidence as it is happening
right now.
I agree this is complex, but we've learned a lot and now have a very good understanding of
the workflow. Let's reach the consensus. I thought we had a silent one because nobody commented
until the patch was submitted. It takes a lot of time and testing, on multiple branches, so
waiting till the last moment is not productive.

> BlockReport retransmissions may lead to storages falsely being declared zombie if storage
report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10301
>                 URL: https://issues.apache.org/jira/browse/HDFS-10301
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.1
>            Reporter: Konstantin Shvachko
>            Assignee: Vinitha Reddy Gankidi
>            Priority: Critical
>         Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, HDFS-10301.004.patch,
HDFS-10301.005.patch, HDFS-10301.006.patch, HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, HDFS-10301.012.patch, HDFS-10301.013.patch,
HDFS-10301.014.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, HDFS-10301.sample.patch,
zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it sends the
block report again. Then NameNode while process these two reports at the same time can interleave
processing storages from different reports. This screws up the blockReportId field, which
makes NameNode think that some storages are zombie. Replicas from zombie storages are immediately
removed, causing missing blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message