hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2742) HA: observed dataloss in replication stress test
Date Wed, 11 Jan 2012 06:42:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183911#comment-13183911

Todd Lipcon commented on HDFS-2742:

bq. What is the implication of ignoring RBW altogether at the standby?

That's an idea I've thought a little about, but I think it has some implications for lease
recovery. In actuality, in order to fix the cases in HDFS-2691, I think we need to send RBW
blockReceived messages to the SBN as soon as a pipeline is constructed.

I do like it, though, as at least a stop-gap for now while we work on a more thorough solution.

bq. If editlog has a finalized record, can we just ignore the RBW from the block report?

Possibly - I haven't thought through the whole Append state machine. I assumed that the code
that marks a RBW replica as corrupt when received for a COMPLETED block is probably there
for a good reason... so changing the behavior there might introduce some other bugs that could
even hurt the non-HA case.

I'm going to keep working on this and see if I can come up with a simpler solution based on
some of Suresh's ideas above.
> HA: observed dataloss in replication stress test
> ------------------------------------------------
>                 Key: HDFS-2742
>                 URL: https://issues.apache.org/jira/browse/HDFS-2742
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node, ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hdfs-2742.txt, log-colorized.txt
> The replication stress test case failed over the weekend since one of the replicas went
missing. Still diagnosing the issue, but it seems like the chain of events was something like:
> - a block report was generated on one of the nodes while the block was being written
- thus the block report listed the block as RBW
> - when the standby replayed this queued message, it was replayed after the file was marked
complete. Thus it marked this replica as corrupt
> - it asked the DN holding the corrupt replica to delete it. And, I think, removed it
from the block map at this time.
> - That DN then did another block report before receiving the deletion. This caused it
to be re-added to the block map, since it was "FINALIZED" now.
> - Replication was lowered on the file, and it counted the above replica as non-corrupt,
and asked for the other replicas to be deleted.
> - All replicas were lost.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message