hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
Date Wed, 14 Mar 2012 21:20:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229634#comment-13229634
] 

Todd Lipcon commented on HDFS-3077:
-----------------------------------

Hi Flavio. The philosophy of having building blocks is a great one. I'm an ardent supporter
of using ZooKeeper, and other building blocks which fit our needs. I think BookKeeper, though,
is like using a chainsaw to get a haircut. It does lots of stuff we don't need, and will take
a ton of work to move it to fit the other requirements we do have.

bq. Also, if these aspects are important for you, why don't you want to contribute them to
the project?

Moving BK over to Hadoop-based configuration, IPC, security, local edit log implementation,
quorum based commit, etc, would leave almost nothing left of BK, except for (a) features we
don't need, like striping and interleaved logs and (b) the core protocol, which you have described
yourself as relatively simple.

bq. Finally, we have focused on the implementation of core protocols like zab and the quorum
consensus of BookKeeper. Why not leverage this experience and focus?
I absolutely am leveraging that experience. I spent some 10 hours this weekend studying your
papers and presentations on BK and ZAB. BK is clearly a very similar system and I'm sure I
will reference its design while working on this system.

Regardless, as I said above, I intend to continue down this avenue. Please continue working
on the BookKeeper one if you think it is a better avenue. If my proposed solution turns into
a disaster because I screw up the quorum implementations, I'm sure no one will use it, and
I'll be glad that you continued to work on an alternate. The joy of open source and pluggable
implementations is that we can both prove our ideas in code and let the community vote with
their feet.

I don't think this argument is likely to be fruitful if we continue. So let's just agree to
disagree and each of us can get to work on proving the other wrong with actual code.
                
> Quorum-based protocol for reading and writing edit logs
> -------------------------------------------------------
>
>                 Key: HDFS-3077
>                 URL: https://issues.apache.org/jira/browse/HDFS-3077
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: ha, name-node
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>
> Currently, one of the weak points of the HA design is that it relies on shared storage
such as an NFS filer for the shared edit log. One alternative that has been proposed is to
depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated
edit log on commodity hardware. This JIRA is to implement another alternative, based on a
quorum commit protocol, integrated more tightly in HDFS and with the requirements driven only
by HDFS's needs rather than more generic use cases. More details to follow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message