hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francis Liu <tof...@apache.org>
Subject Re: DISCUSS: Merge new Assignment Manager (AMv2), HBASE-14614
Date Wed, 24 May 2017 20:56:23 GMT
I'd be up for taking this for a spin. As I need to get split meta rebased on top of this anyway.
Also I'd generally like to know more about AMv2 (transition from zkless etc). BTW once 2.0
is branched does that mean we can't merge in any big features anymore?


    On Wednesday, May 24, 2017 8:37 AM, Stack <stack@duboce.net> wrote:

 On Wed, May 24, 2017 at 8:11 AM, Sean Busbey <busbey@apache.org> wrote:

> Presuming merge happens, looking at the TODOs I see 1 called out
> expressly as a blocker and 2 called out as minor. Is it safe to assume
> the remainder have been evaluated and found some where in between, or
> will we need a triaging step once they're in JIRA to determine number
> of blockers before branch-2 would be ready for release?
> The former.

The item called out as a BLOCKER I have not been unable to reproduce in a
few weeks now. I should probably knock down its priority given this fact
(and that should it happen -- we are talking about dropping Procedures
because of Procedure WAL Corruption -- it will be for hbck to cover until
source of corruption is figured).

I do not know of any other BLOCKERs, not at this point. We want hbase2
rolling upgradeable from hbase1 but that I see as a separate task, one we
might as a community want to punt on, and besides, AMv2, as I see it, would
be a prerequisite.

The others are tasks of some substance, worthy of call out, but for the
most part improvement and finishing.

No harm in me filing JIRAs for the non-nebulous of [1]. Let me work on this

Thanks Sean,


> On Wed, May 24, 2017 at 2:22 AM, Stack <stack@duboce.net> wrote:
> > I would like to discuss merging branch HBASE-14614 to master.
> >
> > The new AssignmentManager on HBASE-14614 branch promises new levels of
> > scaling, insight, resilience and assignment performance. A skeleton dev
> > overview is available here [1].
> >
> > It has been in development a good while now. Currently it is working at
> > least as well as our current assign. Cluster testing has me into a new
> > realm of scale where the failures are for reasons other than AM; e.g.
> > massive WALs that take tens of minutes to split holding up restore post
> > crash (TODO).
> >
> > The AMv2 patch is large, unfortunately. Most of the bulk comes of
> generated
> > protobuf changes but the patch is also large because the commit could not
> > deliver an half-baked machine.
> >
> > The patch is in need of review. Independent confirmation that it is
> > basically working would be sweet.
> >
> > There are outstanding items to complete but they can come in post-merge.
> > Critical is reenabling a set of tests disabled because they depend on
> > workings no longer present or they depended on a broken behavior in AMv1
> > (Disabled tests and the list of TODOs are in [1]).
> >
> > Currently I am working on polish making it pass the outstanding failing
> > unit tests, fixing white space, and findbugs but it is ready for commit.
> > I've put up an RB patch that is absent protobufs so the bulk is less
> > intimidating [3].
> >
> > This feature is the last hurdle to our branching for hbase2.
> >
> > Thanks,
> > St.Ack
> >
> > P.S. This patch is mostly the work of Matteo Bertozzi (inception, bulk of
> > implementation). Others have contributed including Stephen Yuan Jiang.
> > Umesh Agashe and Appy also have contributed fixes that got added to the
> > HBASE-14614 branch.
> >
> > 1.
> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > 2.
> > https://docs.google.com/document/d/1QLXlVERKt5EMbx_
> EL3Y2u0j64FN-_TrVoM5WWxIXh6o/edit#heading=h.df9krsl9k16
> > 3. https://reviews.apache.org/r/55117/

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message