hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
Date Thu, 18 Aug 2011 02:03:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086739#comment-13086739

Jitendra Nath Pandey commented on HDFS-1975:

There are two aspects to this problem. 
# How standby gets the edit logs and reads them?
# How it synchronizes the edit logs with the updates coming from the datanodes?

I am taking following approach.

Access to logs:
# A new configuration parameter will be added, which tells the locations of edit logs that
standby can use to read. This configuration will have subset of locations that are configured
as edit log locations on the primary.
# The standby will use JournalSet implementation of the JournalManager interface to read the
transactions. (Refer HDFS-1580, HDFS-2158).
# If no transactions are available EditLogInputStream throws NoMoreTransactionsException,
the standby just sleeps for a short timeout, and retries again.
# On failover, the standby reads the edit logs for the last time after the primary has been
fenced in following steps:
#* close the edit log input stream.
#* open the edit log input stream.
#* read
#* close
A close in first step could be useful since edit log is stored on nfs, to make sure that latest
changes are visible.

Updates from datanodes
Standby maintains latest generation stamp, based on the records from the editlog. This is
used for processing updates from datanodes as follows:
# The standby receives block report/block received/block deleted (pending HDFS-395) message
from the datanode.
#* If the block's GS is less than the current GS of the standby, the standby will process
#* If the block's GS is greater than the current GS of the standby, the standby will buffer
this block received/deleted and waits until it sees corresponding generation stamp in the
edit log. Otherwise it processes the block received.

> HA: Support for sharing the namenode state from active to standby.
> ------------------------------------------------------------------
>                 Key: HDFS-1975
>                 URL: https://issues.apache.org/jira/browse/HDFS-1975
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>            Reporter: Suresh Srinivas
>            Assignee: Jitendra Nath Pandey
>         Attachments: hdfs-1975.txt, hdfs-1975.txt
> To enable hot standby namenode, the standby node must have current information for -
namenode state (image + edits) and block location information. This jira addresses keeping
the namenode state current in the standby node. To do this, the proposed solution in this
jira is to use a shared storage to store the namenode state. 
> Note one could also build an alternative solution by augmenting the backup node. A seperate
jira could explore this.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message