hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3425) Enable standby namenode to tail edits from journal service
Date Tue, 19 Jun 2012 01:02:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396431#comment-13396431

Todd Lipcon commented on HDFS-3425:

bq. I am not sure I got the idea about using selectInputStreams() instead of if(usesJournalService()).
Is your branch currently visible, so we can see an example?

That's what I'm working on right now. But basically, the idea is the following:

- the NN runs with just a QuorumJournalManager, which maintains the list of remote JournalNodes
to talk to. I believe your design doc calls this a JournalList instead, but same thing.
- when the NN (or the Standby NN) wants to read edits, it uses editLog.selectInputStreams()
exactly as it does today. But the QuorumJournalManager implementation of it services the call
as follows:
-- RPC to each of the remote journals
-- get back a list of finalized edit log segments
-- for each, creates an object which stores the HTTP URL at which that segment is accessible,
along with the txid info
-- creates an EditLogFileInputStream for each such object, but when opened, instead of creating
FileInputStream, it opens the URL connection and calls getInputStream() on the resulting URL

So, the NN code remains identical, and all the magic is encapsulated in the journal manager
implementation. This is basically the same as how the BookKeeper implementation works today
-- the journal manager just has to implement selectInputStreams and the rest of the code _should_
work the same without special cases.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>                 Key: HDFS-3425
>                 URL: https://issues.apache.org/jira/browse/HDFS-3425
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>            Reporter: Brandon Li
>            Assignee: Brandon Li
>         Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
> In an HA cluster with the journal service configured, namenode should get edit logs from
journal service instead of from local or shared storage directories.  
> As one of the first steps towards this goal, this JIRA is to enable standby namenode
to tail edits from journal services. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message