hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
Date Tue, 03 Apr 2012 14:24:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245346#comment-13245346

Suresh Srinivas commented on HDFS-3092:

bq. Your design does not seem to consider the possibility of multiple concurrent logs, which
you may want to have for federation. 
For HDFS editlogs, my feeling is that there will only be three JDs. One on the active namenode,
second on the standby and a third JD on one of the machines. In federation, one has to configure
a JD per Federated namespace. Alternative is to use BookKeeper, since it could make the deployment
simpler for federated large cluster.

bq. There has been comments about comparing the different approaches discussed, and I was
wondering what criteria you have been thinking of using to compare them.
I think the comment was more about comparing the design and complexity of deployment and not
benchmarks for two systems. Performance is not the motivation for this jira.

bq. I was wondering about how reads to the log are executed if writes only have to reach a
majority quorum. Once it is time to read, how does the reader gets a consistent view of the
log? One JD alone may not have all entries, so I suppose the reader may need to read from
multiple JDs to get a consistent view? Do the transaction identifiers establish the order
of entries in the log? One quick note is that I don't see why a majority is required; bk does
not require a majority.
We decided on majority quorum to keep the design simple, though it is strictly not necessary.
A JD in JournalList is supposed to have all the entries and any JD from the list can be used
to read the journals.

bq. Here are some notes I took comparing the bk approach with the one in this jira, in the
case you're interested
I noticed that as well. After we went thourgh many issues that this solution had to take care
of, the solution looks very similar to BK. That is comforting :-)

> Enable journal protocol based editlog streaming for standby namenode
> --------------------------------------------------------------------
>                 Key: HDFS-3092
>                 URL: https://issues.apache.org/jira/browse/HDFS-3092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, name-node
>    Affects Versions: 0.24.0, 0.23.3
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: MultipleSharedJournals.pdf
> Currently standby namenode relies on reading shared editlogs to stay current with the
active namenode, for namespace changes. BackupNode used streaming edits from active namenode
for doing the same. This jira is to explore using journal protocol based editlog streams for
the standby namenode. A daemon in standby will get the editlogs from the active and write
it to local edits. To begin with, the existing standby mechanism of reading from a file, will
continue to be used, instead of from shared edits, from the local edits.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message