hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Junqueira (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
Date Fri, 20 Apr 2012 21:21:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258594#comment-13258594

Flavio Junqueira commented on HDFS-3092:

Thanks for posting this comparison, Bikas. Let me try to address the last two points on bookkeeper:

bq. Tools for recovery - have a bookie recover tool. others?? 

That's correct, we have a bookie recovery tool that reconstructs the ledger fragments of a
dead bookie. This has been part of bookkeeper for a while. We have some other tools proposed
in BOOKKEEPER-183 to read and check bookie files, but they are not checked in yet. We have
yet some other tools we want to develop for some more extreme failure scenarios. We are targeting
release 4.2.0 for them (a draft of our feature roadmap is here https://cwiki.apache.org/confluence/display/BOOKKEEPER/Roadmap).

bq. Release frequency, committers, projects that use it??

We started planning for releases every 6 months, but we have been thinking about releasing
more frequently, every 3 months. 

We are currently 6 committers, but only 3 have been really active. Four of us are from Yahoo!,
one from Twitter, and one from Facebook. Given that it is still a young project, I don't see
why other hdfs folks cannot become committers of bookkeeper if they contribute and there is
interest. It would be actually quite natural in the case bookkeeper ends up being used with
the namenode. For us, having committers from the hdfs community would be useful to make sure
we don't miss important requirements of yours.

As for projects using it, we have applications that incorporated bookkeeper (and hedwig) inside
Yahoo! recently, and we have people from other companies on the mailing list discussing their
setups and asking questions. If you're on the list, you have possibly seen those.

> Enable journal protocol based editlog streaming for standby namenode
> --------------------------------------------------------------------
>                 Key: HDFS-3092
>                 URL: https://issues.apache.org/jira/browse/HDFS-3092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, name-node
>    Affects Versions: 0.24.0, 0.23.3
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: ComparisonofApproachesforHAJournals.pdf, MultipleSharedJournals.pdf,
MultipleSharedJournals.pdf, MultipleSharedJournals.pdf
> Currently standby namenode relies on reading shared editlogs to stay current with the
active namenode, for namespace changes. BackupNode used streaming edits from active namenode
for doing the same. This jira is to explore using journal protocol based editlog streams for
the standby namenode. A daemon in standby will get the editlogs from the active and write
it to local edits. To begin with, the existing standby mechanism of reading from a file, will
continue to be used, instead of from shared edits, from the local edits.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message