hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1972) HA: Datanode fencing mechanism
Date Tue, 13 Dec 2011 18:30:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168577#comment-13168577

dhruba borthakur commented on HDFS-1972:

> If an invalidateBlock call has to be postponed, add that block to a blocksToRecheck list
of some kind.
When all of the DNs have become trusted, scan all of the blocks in blocksToRecheck checking
for misreplication.

Suppose we do not keep a blocksToRecheck list.  When any DN flips its trusted flag to true,
check if all of the DNs are now trusted. If this is the case, go through the processMisReplicatedBlocks
code path to scan all blocks in the system for potential over-replication, adding them to
the invalidate queue. The execution of the  processMisReplicatedBlocks() can occur in a background
thread... better still, chooseExcessReplicates() does not really need the FSNamesystem lock.
We have done this just to increase scalability: http://bit.ly/rUDVui

Just a thought to not increase yet another data structure 'blocksToRecheck', but if you feel
that this is absolutely necessary, I am fine with it.

> HA: Datanode fencing mechanism
> ------------------------------
>                 Key: HDFS-1972
>                 URL: https://issues.apache.org/jira/browse/HDFS-1972
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node, ha, name-node
>            Reporter: Suresh Srinivas
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1972-v1.txt, hdfs-1972.txt
> In high availability setup, with an active and standby namenode, there is a possibility
of two namenodes sending commands to the datanode. The datanode must honor commands from only
the active namenode and reject the commands from standby, to prevent corruption. This invariant
must be complied with during fail over and other states such as split brain. This jira addresses
issues related to this, design of the solution and implementation.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message