Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates
 209.85.215.48 as permitted sender)
Received-SPF: pass (google.com: domain of todd@cloudera.com designates
 10.112.42.72 as permitted sender) client-ip=10.112.42.72;
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s6mQEQcLH3TEpmr6EQwEz81y9t2NbVFY3GRpAv55AqE6A@mail.gmail.com>
References: 
 <CA+4052nLZ7sindNRADaAOXnU=XK79A1DYH==PJOKJA84v=a5UA@mail.gmail.com>
 <CADdVdVFP-oVbAT6oz6UdQ6KAZkJSGJPkPCAgPo5hfExgcCBrxQ@mail.gmail.com>
 <CADY20s6mQEQcLH3TEpmr6EQwEz81y9t2NbVFY3GRpAv55AqE6A@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 23 Feb 2012 18:08:22 -0800
Message-ID: 
 <CADY20s6SUFW=eYX8eW81-gCMtzSed=-Kmc+M7nME2KGmnSLNFQ@mail.gmail.com>
Subject: Re: Merging the HA branch to trunk - Wednesday, February 29th
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi folks,

I had the chance today to run some performance benchmarks on one of
Cloudera's 100 node test clusters. I've posted the results to the
JIRA, but the summary is that I think we need to do a bit more
optimization of the persistBlocks edit log entries before we merge.

That said, I think there are a couple easy wins that should bring
performance back in line with trunk, which I expect to complete by
early next week.

Please continue reviewing the branch so that when the optimizations
have been made, we can proceed with a merge.

-Todd

On Thu, Feb 23, 2012 at 10:27 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Feb 23, 2012 at 10:23 AM, Suresh Srinivas
> <suresh@hortonworks.com> wrote:
>> I am not sure any of these issues are serious show stoppers for merging
>> into trunk.
>> Why not merge into trunk and fix some of these issues?
>>
>> The reason is, merging is non trivial with two branches changing
>> independently. Given that
>> Jitendra has posted a merge patch, why not do it earlier? Do we need heads
>> up of a week.
>> If merging must wait, should we consider creating a merge branch and
>> committing the patch
>> Jitendra has. This makes other merges more manageable.
>
> We already have a merge branch - the patch is easy to generate since
> we have been merging in the trunk->HA direction daily since its
> inception.
>
> I'm all for merging to trunk earlier if everyone's cool with it, but
> we do need to start a vote. Shall I call one?
>
> -Todd
>
>> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <atm@cloudera.com> wrote:
>>
>>> Hello HDFS devs,
>>>
>>> Work has largely stabilized on the HA-branch in the last few weeks. At this
>>> point the HA NN project is nearly feature-complete for manual failover.
>>> We've been running the full test suite nightly, and all automated tests
>>> have been passing, except for one known test failure which should be fixed
>>> shortly.
>>>
>>> I'd like to begin the process of merging this branch back to HDFS trunk.
>>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
>>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
>>> relate to supporting automatic failover. Once the branch is merged to
>>> trunk, work on these JIRAs can continue there.
>>>
>>> I've identified the following JIRAs which I think should be the only
>>> remaining blockers for merging to trunk:
>>>
>>> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
>>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
>>> stuff.
>>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go through
>>> failover path
>>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
>>> (will fix the only current test failure)
>>>
>>> All of these JIRAs should be fixed in the next few days.
>>>
>>> I propose that, unless more blocker issues are discovered in the interim,
>>> we merge this branch to trunk one week from today, i.e. Wednesday, February
>>> 29th. During this time we will also execute the test plans described in the
>>> test documents attached to HDFS-1623 to try to identify any regressions or
>>> performance issues in the branch. If you plan to review the code changes or
>>> the test plan, I ask that you please do so as soon as possible.
>>>
>>> Feedback is certainly welcome on this plan.
>>>
>>> Thanks a lot,
>>> Aaron
>>>
>>> --
>>> Aaron T. Myers
>>> Software Engineer, Cloudera
>>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera


-- 
Todd Lipcon
Software Engineer, Cloudera