hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
Date Wed, 14 Sep 2016 03:13:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15489223#comment-15489223
] 

Arpit Agarwal commented on HDFS-10301:
--------------------------------------

bq. I have modified the checkLease method in BlockReportLeaseManager to return true when node.leaseId
== 0. Please let me know if you see any issues with this approach.
[~redvine], IIUC this workaround bypasses the lease checks but the last report detection logic
still remains broken. I am no longer sure zombie removal in heartbeats is safe and I was probably
wrong to add it in HDFS-7596. Zombie removal is safe just after processing all storage reports
from a full block report. So I think we should fix "last report detection".

I believe the following two changes will fix this problem (same suggestion as my previous
comment):
# The DataNode sends a flag with the last RPC message that indicates all the previous reports
have been successfully processed. This is guaranteed to be correct and removes the burden
from the NN.
# Eliminate single-RPC reports as Daryn suggested.

Any thoughts on this?

Thanks Konstantin and Vinitha for reporting this problem and your marathon efforts to fix
it. It is a hard problem so I request we aim for consensus before committing a fix. 

> BlockReport retransmissions may lead to storages falsely being declared zombie if storage
report processing happens out of order
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10301
>                 URL: https://issues.apache.org/jira/browse/HDFS-10301
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.1
>            Reporter: Konstantin Shvachko
>            Assignee: Vinitha Reddy Gankidi
>            Priority: Critical
>             Fix For: 2.7.4
>
>         Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, HDFS-10301.004.patch,
HDFS-10301.005.patch, HDFS-10301.006.patch, HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch,
HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, HDFS-10301.012.patch, HDFS-10301.013.patch,
HDFS-10301.014.patch, HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, HDFS-10301.sample.patch,
zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it sends the
block report again. Then NameNode while process these two reports at the same time can interleave
processing storages from different reports. This screws up the blockReportId field, which
makes NameNode think that some storages are zombie. Replicas from zombie storages are immediately
removed, causing missing blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message