hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
Date Mon, 02 Apr 2012 17:07:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244341#comment-13244341

Todd Lipcon commented on HDFS-3092:

Hi Suresh. I took a look at the design document, and I think it actually shares a lot with
what I'm doing in HDFS-3077. Hopefully we can share some portions of the code and design.

Here are some points I think need elaboration in the design doc:
- How does the fencing command ensure that prior NNs can no longer access the JD after it
completes? I think you need to have the JDs record the sequence number of each NN so they
can reject past NNs from coming back to life.
- I don't think the following can be done correctly:
a. Choose the transaction >= Q JDs from JournalList.
b. Else choose the highest transaction ID from a JD in the JournalList.
3. All the JDs perform recovery to transaction ID Ft .
because it's possible that there are distinct transactions with the same txid at the beginning
of a log segment. For example, consider the following situation with two NN (NN1 and NN2)
and three JDs (JD1, JD2, JD3):

1. NN1 writes txid 1 to JD1, crashes before writing to JD2 and JD3 
2. NN2 initiates fencing, but only succeeds in contacting JD2 and JD3. So, it does not see
the edit made in step 1
3. NN2 writes txid1 to JD2 and JD3, then crashes
4. One of the two NNs recovers. It sees that all JDs have txid 1. The fencing/synchronization
process you've described cannot distinguish between the correct txn which was written to a
quorum and the incorrect txn which was only written to JD1.
> Enable journal protocol based editlog streaming for standby namenode
> --------------------------------------------------------------------
>                 Key: HDFS-3092
>                 URL: https://issues.apache.org/jira/browse/HDFS-3092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, name-node
>    Affects Versions: 0.24.0, 0.23.3
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: MultipleSharedJournals.pdf
> Currently standby namenode relies on reading shared editlogs to stay current with the
active namenode, for namespace changes. BackupNode used streaming edits from active namenode
for doing the same. This jira is to explore using journal protocol based editlog streams for
the standby namenode. A daemon in standby will get the editlogs from the active and write
it to local edits. To begin with, the existing standby mechanism of reading from a file, will
continue to be used, instead of from shared edits, from the local edits.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message