hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
Date Fri, 27 May 2016 20:22:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304702#comment-15304702

Colin Patrick McCabe commented on HDFS-10301:

[~redvine], the fact that you are having trouble with stale storages versus zombie storages
is because your patch uses a separate mechanism to detect what storages exist on the DN. 
The existing code doesn't have this problem because the full block report itself acted as
the record of what storages existed.  This is one negative side effect of the more complex
approach.  Another negative side effect is that you are transmitting the same information
about which storages are present multiple times.

Despite these negatives, I'm still willing to review a patch that uses the more complicated
method as long as you don't introduce extra RPCs.  I agree that we should remove a stale storage
if it doesn't appear in the full listing that gets sent.  Just to be clear, I am -1 on a patch
which adds extra RPCs.  Perhaps you can send this listing in an optional field in the first

[~daryn], I don't like the idea of "band-aiding" this issue rather than fixing it at the root.
 Throwing an exception on interleaved storage reports, or forbidding combined storage reports,
seem like very brittle work-arounds that could easily be undone by someone making follow-on
changes.  I came up with patch 005 and the earlier patches as a very simple fix that could
easily be backported.  If you are interested in something simple, then please check it out...
or at least give a reason for not checking it out.

> BlockReport retransmissions may lead to storages falsely being declared zombie if storage
report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>                 Key: HDFS-10301
>                 URL: https://issues.apache.org/jira/browse/HDFS-10301
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.1
>            Reporter: Konstantin Shvachko
>            Assignee: Colin Patrick McCabe
>            Priority: Critical
>         Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, HDFS-10301.004.patch,
HDFS-10301.005.patch, HDFS-10301.01.patch, HDFS-10301.sample.patch, zombieStorageLogs.rtf
> When NameNode is busy a DataNode can timeout sending a block report. Then it sends the
block report again. Then NameNode while process these two reports at the same time can interleave
processing storages from different reports. This screws up the blockReportId field, which
makes NameNode think that some storages are zombie. Replicas from zombie storages are immediately
removed, causing missing blocks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message