hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Srinivas <sur...@hortonworks.com>
Subject Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk
Date Tue, 09 Oct 2012 03:32:56 GMT
On Mon, Oct 8, 2012 at 8:03 PM, Andrew Purtell <apurtell@apache.org> wrote:

> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
> we don't have a significant stake in this discussion except at a meta level
> as potential contributors.

Andrew, as I had said in person to you, thank you for helping in testing
feature. This is very useful for the community.

> This is a case study of why feature branch development is problematic? A
> would-be contributor can make a significant effort over months and end up
> with a baked result, ready to move on to a perhaps large backlog of other
> work. Others can reasonably be expected meanwhile to triage their attention
> until the merge point. Then, reconsidering design points will be more
> challenging than a design discussion at an earlier time, because there is
> already a significant sunk cost in the current design. What does a feature
> branch buy here over development in situ in trunk (like the BookKeeper JM)?
> Should would-be future contributors consider a feature branch as a viable
> route toward contribution or not?

Andrew, feature branches have worked quite well in HDFS. See many of the
feature developments that have happened in different branches. Let me
again. I am not trying to block this work.

I have explained the reasons why merge time made sense for this jira, given
the complexity of design,
where it started with ZAB, then added paxos in the middle of the
These protocols are quite complicated, with multiple papers published on
protocol, the proof, how to implement it and various simplifications and
variations of the protocol.
Requiring intimate knowledge of this and applying it to namenode journals
is non
trivial. I and Sanjay have spent many hours every day last three weeks on
understanding this.
Not all features have this complicated code and protocols and hence do not
require this
level of review.

All I am suggesting here is that we wait for the design discussion to
complete. Based on my comments or Sanjay's comment, you would see that
no one is asking for complete redesign. Some of the subtle issues and their
solutions may not be clear to all. See the discussions in the jira and
compare it with
what is in the design. Second, based on the protocols used, the
questions that are being asked are, can step x and step y be merged. Other
are around perceived diversions from the well known protocols and its
(example ZooKeeper).

At the end of these discussion there may be few or no changes required.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message