Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates
 74.125.82.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s4RLa6XbkepHSo=eHyZjMB6AepSQdiQMuQ0r+gX3NsOew@mail.gmail.com>
References: 
 <CADY20s544EPrgFoY8aF15nDHbupSW_rvxt8SWVdRA6KmYogxRg@mail.gmail.com>
	<CAPn_vTvYh6TuCV1zvRqKQ-CrwM1WJ+mGg=osqhVW-eOi3vZwdw@mail.gmail.com>
	<CAKtuutGjDw3kYy6pvOTxgv1ZRViAY7dOd9-_j+7AiUEfz9BDAA@mail.gmail.com>
	<CADY20s4RLa6XbkepHSo=eHyZjMB6AepSQdiQMuQ0r+gX3NsOew@mail.gmail.com>
Date: Wed, 26 Sep 2012 10:54:32 -0700
Message-ID: 
 <CAPn_vTsZAkBKH8yUP73kCEpNRvq4cW39RHeHAxuxEM-RmzZM-Q@mail.gmail.com>
Subject: Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk
From: Eli Collins <eli@cloudera.com>
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Sep 26, 2012 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
>> I think this is a great work, Todd.
>> And I think we should not merge it into trunk or other branches.
>> As I suggested earlier on this list I think this should be spinned off
>> as a separate project or a subproject.
>>
>> - The code is well detached as a self contained package.
>
> The addition is mostly self-contained, but it makes use of a bunch of
> "private" parts of HDFS and Common:
> - Reuses all of the Hadoop security infrastructure, IPC, metrics, etc
> - Coupled to the JournalManager interface which is still evolving. In
> fact there were several patches in trunk which were done during the
> development of this project, specifically to make this API more
> general. There's still some further work to be done in this area on
> the generic interface -- eg support for upgrade/rollback.
> - The functional tests make use of a bunch of "private" HDFS APIs as well.
>
>> - It is a logically stand-alone project that can be replaced by other
>> technologies.
>> - If it is a separate project then there is no need to port it to
>> other versions. You can package it as a dependent jar.
>
> Per above, it's not that separate, because in order to build it, we
> had to make a number of changes to core HDFS internal interfaces. It
> currently couldn't be used to store anything except for NN logs. It
> would be a nice extension to truly separate it out into a
> content-agnostic quorum-based edit log, but today it actually uses the
> existing edit log validation code to determine valid lengths, etc.
>
>> - Finally, it will be a good precedent of spinning new projects out of
>> HDFS rather than bringing everything under HDFS umbrella.
>>
>> Todd, I had a feeling you were in favor of this direction?
>
> I'm not in favor of it - I had said previously that it's worth
> discussing if several other people believe the same.

I'm not in favor of it either. One of the benefits of QJM over the BK
approach is that it's embedded in HDFS (ie not treated as a separate
storage system). See HDFS-3077 and HDFS-3092 for details on that
discussion.