hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Merging the HA branch to trunk - Wednesday, February 29th
Date Fri, 24 Feb 2012 02:08:22 GMT
Hi folks,

I had the chance today to run some performance benchmarks on one of
Cloudera's 100 node test clusters. I've posted the results to the
JIRA, but the summary is that I think we need to do a bit more
optimization of the persistBlocks edit log entries before we merge.

That said, I think there are a couple easy wins that should bring
performance back in line with trunk, which I expect to complete by
early next week.

Please continue reviewing the branch so that when the optimizations
have been made, we can proceed with a merge.


On Thu, Feb 23, 2012 at 10:27 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Feb 23, 2012 at 10:23 AM, Suresh Srinivas
> <suresh@hortonworks.com> wrote:
>> I am not sure any of these issues are serious show stoppers for merging
>> into trunk.
>> Why not merge into trunk and fix some of these issues?
>> The reason is, merging is non trivial with two branches changing
>> independently. Given that
>> Jitendra has posted a merge patch, why not do it earlier? Do we need heads
>> up of a week.
>> If merging must wait, should we consider creating a merge branch and
>> committing the patch
>> Jitendra has. This makes other merges more manageable.
> We already have a merge branch - the patch is easy to generate since
> we have been merging in the trunk->HA direction daily since its
> inception.
> I'm all for merging to trunk earlier if everyone's cool with it, but
> we do need to start a vote. Shall I call one?
> -Todd
>> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <atm@cloudera.com> wrote:
>>> Hello HDFS devs,
>>> Work has largely stabilized on the HA-branch in the last few weeks. At this
>>> point the HA NN project is nearly feature-complete for manual failover.
>>> We've been running the full test suite nightly, and all automated tests
>>> have been passing, except for one known test failure which should be fixed
>>> shortly.
>>> I'd like to begin the process of merging this branch back to HDFS trunk.
>>> There are still several outstanding sub-JIRAs under the HDFS-1623 and
>>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or
>>> relate to supporting automatic failover. Once the branch is merged to
>>> trunk, work on these JIRAs can continue there.
>>> I've identified the following JIRAs which I think should be the only
>>> remaining blockers for merging to trunk:
>>> HDFS-2904 - Client support for getting delegation tokens in an HA cluster
>>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup
>>> stuff.
>>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go through
>>> failover path
>>> HDFS-2979 - Balancer should use logical URI for creating failover proxy
>>> (will fix the only current test failure)
>>> All of these JIRAs should be fixed in the next few days.
>>> I propose that, unless more blocker issues are discovered in the interim,
>>> we merge this branch to trunk one week from today, i.e. Wednesday, February
>>> 29th. During this time we will also execute the test plans described in the
>>> test documents attached to HDFS-1623 to try to identify any regressions or
>>> performance issues in the branch. If you plan to review the code changes or
>>> the test plan, I ask that you please do so as soon as possible.
>>> Feedback is certainly welcome on this plan.
>>> Thanks a lot,
>>> Aaron
>>> --
>>> Aaron T. Myers
>>> Software Engineer, Cloudera
> --
> Todd Lipcon
> Software Engineer, Cloudera

Todd Lipcon
Software Engineer, Cloudera

View raw message