hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3425) Enable standby namenode to tail edits from journal service
Date Tue, 19 Jun 2012 01:45:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396447#comment-13396447

Todd Lipcon commented on HDFS-3425:

Hey Brandon,

I just got my implementation "working" (ie I can run a NN against remote loggers and then
successfully restart reading the edits remotely). It's nowhere near finished/tested/pretty,
but I pushed to github so I could share the approach:

This commit https://github.com/toddlipcon/hadoop-common/commit/c7aeeb83a653935463d0899968828abad3597ee8
refactors EditLogFileInputStream to deal with an interface instead of directly a {{File}}
reference. The interface just has to be able to provide an input stream as well as its length
and a name.

This commit https://github.com/toddlipcon/hadoop-common/commit/2969ab41478c808ff26ef3f00dad2cd22a352e93
adds an implementation of the above interface which reads from a remote URL, and changes the
getEditLogManifest call to also expose the node's HTTP port (so it can use ephemeral ports)

Then this commit https://github.com/toddlipcon/hadoop-common/commit/477d937c258d4ffe67a5042258da7410df74c42e
implements {{selectInputStreams()}} to actually pull the edits off the remote nodes while
loading the namespace.

I'm not attacking HA yet in these patches, but the interfaces should work out the same there,
given the BK folks managed to get it working.

As I said above, the code's rough, so I don't expect anyone to code review the github branch,
but if you have questions on the approach, let me know. I'm trying to reuse as much of your
JournalService code as possible, but mostly doing so by copy-pasting into these new files
for now, to avoid conflicts, etc. I anticipate running back over it to unify the code better
at a later date.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>                 Key: HDFS-3425
>                 URL: https://issues.apache.org/jira/browse/HDFS-3425
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>            Reporter: Brandon Li
>            Assignee: Brandon Li
>         Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
> In an HA cluster with the journal service configured, namenode should get edit logs from
journal service instead of from local or shared storage directories.  
> As one of the first steps towards this goal, this JIRA is to enable standby namenode
to tail edits from journal services. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message