hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Drob <md...@apache.org>
Subject Re: Moving 2.0 forward
Date Fri, 21 Jul 2017 21:02:27 GMT
Yea, beta sounds good to me. Probably we'll find more things once we start
running ITBLL and friends for long periods, but that can maybe come later.
Existing tests should be cleaned up "soon".

On Fri, Jul 21, 2017 at 3:45 PM, Andrew Purtell <apurtell@apache.org> wrote:

> Beta?
>
>
> On Fri, Jul 21, 2017 at 1:16 PM, Stack <stack@duboce.net> wrote:
>
> > On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <mdrob@apache.org> wrote:
> >
> > >
> > > One thing that is not clear to me from your summary and the linked
> > document
> > > is which release we can block on tests? Unit tests are in bad shape
> (but
> > > getting better) and I recently started looking at ITs which also need
> > some
> > > care.
> > >
> > >
> > Dunno. What you think boss? I'd say we can't release if tests are a mess.
> > Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
> > no beta unless all tests (unit and IT) are passing.
> >
> > For sure testing -- as in exercising new features, changed configs, and
> > just straight longevity under stress -- is badly wanting. I should write
> up
> > a matrix I suppose? Let me work on it.
> >
> > Thanks Mike,
> > S
> >
> >
> >
> >
> > > Mike
> > >
> > > On Fri, Jul 21, 2017 at 11:03 AM, Stack <stack@duboce.net> wrote:
> > >
> > > > Status update girls and boys!
> > > >
> > > > hbase-2.0.0-alpha1 went out June 22nd.
> > > >
> > > > alpha2 has been a bit slow to follow (holidays) though there has been
> > > > steady progress closing out blockers and criticals by a bunch of you
> > all.
> > > > The plan is for a release in the first week or so of August. It
> should
> > be
> > > > fully up on hbase-thirdparty using updated (and relocated) versions
> of
> > > > netty, guava, and protobuf as well as a default deploy that has
> > > > master-carrying-no-regions.
> > > >
> > > > alpha3 will follow soon after and will focus on making sure our
> > > user-facing
> > > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> > on)
> > > > and that basic upgrade 'works'.
> > > >
> > > > betas start in September?
> > > >
> > > > I've been keeping a rough general state here [1] (please update any
> > > section
> > > > that is lagging actuality) but for details on what blockers and
> > criticals
> > > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> > 2.0
> > > > into better focus. Feel free to review and punt items you think can
> > wait
> > > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > 1.
> > > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > ktczrlKHK8N4SZzs/edit#
> > > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > > >
> > > >
> > > > On Mon, May 15, 2017 at 4:05 PM, Stack <stack@duboce.net> wrote:
> > > >
> > > > >
> > > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <
> > palomino219@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> HBASE-18037 is a new blocker. I'm currently working on it, will
be
> > > > >> finished
> > > > >> soon I think.
> > > > >>
> > > > >> I made it a blocker then and added it to our hbase2 release doc
> [1]
> > > list
> > > > > as a blocker.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > > ktczrlKHK8N4SZzs/edit#
> > > > >
> > > > >
> > > > >> 2017-05-15 14:12 GMT+08:00 Stack <stack@duboce.net>:
> > > > >>
> > > > >>
> > > > >> > A month on. Status.
> > > > >> >
> > > > >> > I've been working on the HBASE-14614 branch cluster testing.
> > After a
> > > > >> load
> > > > >> > of fixing, the branch passes smaller test runs (an hour
or so of
> > > ITBLL
> > > > >> up
> > > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale
> I've
> > > not
> > > > >> done
> > > > >> > in a while, I start to run into other interesting issues
-- some
> > of
> > > > >> which
> > > > >> > are related to AMv2 (I'm fixing), but others are not (100G
WALs
> > that
> > > > >> take
> > > > >> > ten minutes to split makes for interesting cascades when
monkeys
> > > kill
> > > > >> > inside the ten minutes...). I intend to keep on with this
larger
> > > scale
> > > > >> > testing since it is uncovering good stuff (especially when
HDFS
> is
> > > dog
> > > > >> slow
> > > > >> > because of background replications) but my thinking is that
I
> > should
> > > > be
> > > > >> > large scale testing branch-2, not just HBASE-14614. I think
> > > > HBASE-14614,
> > > > >> > the new AMv2, is good enough to merge to master these times.
> Given
> > > it
> > > > is
> > > > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > > > >> >
> > > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in
the
> next
> > > day
> > > > >> or so
> > > > >> > (I need to fix some unit tests...).
> > > > >> >
> > > > >> > The AMv2 doc is still a work in progress but should give
a gist
> on
> > > > >> where we
> > > > >> > are currently[1].  There is a bunch of todo still but seems
> > > tractable;
> > > > >> e.g.
> > > > >> > rolling upgrade, finish doc., and we don't have an HBCK
since it
> > > needs
> > > > >> to
> > > > >> > be recast in light of how stuff now works but a redo on
HBCK is
> > > > >> premature
> > > > >> > given we don't know failure types as yet (we just fix the
> problems
> > > as
> > > > >> they
> > > > >> > come up).
> > > > >> >
> > > > >> > St.Ack
> > > > >> > 1.
> > > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <stack@duboce.net>
> wrote:
> > > > >> >
> > > > >> > > Some status:
> > > > >> > >
> > > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat
my
> > > disabling
> > > > of
> > > > >> > all
> > > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect
that
> > they
> > > > can
> > > > >> > move
> > > > >> > > hbase;meta off master (AMv2 enforces this constraint;
it is
> > > supposed
> > > > >> to
> > > > >> > be
> > > > >> > > enforced on AMv1 but meta-on-master is incompletely
realized
> in
> > > AMv1
> > > > >> and
> > > > >> > > AMv2). A few other tests have been disabled for various
> reasons.
> > > See
> > > > >> [1]
> > > > >> > > for full list.
> > > > >> > >
> > > > >> > > There is a hefty list of TODOs still (Again see the
messy doc
> > [1])
> > > > but
> > > > >> > the
> > > > >> > > only 'blocker', IMO, is community confidence in AMv2.
> Currently,
> > > > >> cluster
> > > > >> > > tests with chaos fail (new form of 'stuck' regions).
Takes
> time
> > > > >> > > investigating.
> > > > >> > >
> > > > >> > > Will keep you all posted.
> > > > >> > > St.Ack
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > > > apurtell@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> +1 on branching (yay!)
> > > > >> > >>
> > > > >> > >> I have EC2 resources for running ITBLL etc.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <stack@duboce.net>
> > wrote:
> > > > >> > >>
> > > > >> > >> > Some notes on progress toward hbase2.
> > > > >> > >> >
> > > > >> > >> > Given that stability and performance are NOT
emergent
> > behaviors
> > > > but
> > > > >> > >> rather
> > > > >> > >> > projects unto themselves, my thought is that
we commit all
> > that
> > > > >> we've
> > > > >> > >> > agreed as core for hbase2 (see [1]), branch,
and then work
> on
> > > > >> > >> stabilizing
> > > > >> > >> > and perf rather than do stabilize, commit,
and then branch.
> > > What
> > > > >> this
> > > > >> > >> means
> > > > >> > >> > in practice is that for features like Inmemory
Compaction,
> we
> > > > >> commit
> > > > >> > it
> > > > >> > >> > defaulted 'on' ("BASIC" mode) which is what
we want in
> > hbase2.
> > > > >> Should
> > > > >> > it
> > > > >> > >> > prove problematic under test, we disable it
before release.
> > > > >> > >> >
> > > > >> > >> > Are folks good w/ this mode? I ask because,
in a few issues
> > > there
> > > > >> are
> > > > >> > >> > requests for proof that a master feature is
'stable' before
> > > > commit.
> > > > >> > >> This is
> > > > >> > >> > normally a healthy request only in master's
case, it is
> hard
> > to
> > > > >> > >> demonstrate
> > > > >> > >> > stability given its current state.
> > > > >> > >> >
> > > > >> > >> > Other outstanding issues such as decisions
about whether
> > master
> > > > >> hosts
> > > > >> > >> > system tables only (by default), I'm thinking,
we can work
> > out
> > > > post
> > > > >> > >> branch
> > > > >> > >> > in alpha/betas before release.
> > > > >> > >> >
> > > > >> > >> > The awkward item is the long-pole Assignment
Manager. This
> is
> > > an
> > > > >> > >> > all-or-nothing affair. Here we are switching
in a new
> Master
> > > > core.
> > > > >> > >> While I
> > > > >> > >> > think it fine that AMv2 is incomplete come
branch time,
> those
> > > of
> > > > us
> > > > >> > >> working
> > > > >> > >> > on the new AM still need to demonstrate to
you all that it
> > > > >> basically
> > > > >> > >> > viable.
> > > > >> > >> >
> > > > >> > >> > The point-of-no-return is commit of the patch
in
> HBASE-14614.
> > > > >> > >> HBASE-14614
> > > > >> > >> > (AMv2) is coming close to passing all unit
tests. We'll
> spend
> > > > some
> > > > >> > time
> > > > >> > >> > running it on a cluster to make sure it fundamentally
sound
> > and
> > > > >> will
> > > > >> > >> report
> > > > >> > >> > back on our experience. There has been an
ask for some dev
> > doc
> > > > and
> > > > >> > >> > low-levels on how it works (in progress).
Let satisfaction
> of
> > > > these
> > > > >> > >> > requests be blockers on commit. We'll put
the HBASE-14614
> > > commit
> > > > up
> > > > >> > for
> > > > >> > >> a
> > > > >> > >> > vote on dev list given its import.
> > > > >> > >> >
> > > > >> > >> > Branch will happen after HBASE-14614 goes
in (or its
> > rejection)
> > > > >> with
> > > > >> > our
> > > > >> > >> > first alpha soon after. Its looking like a
week or two at
> > least
> > > > >> given
> > > > >> > >> how
> > > > >> > >> > things have been going up to this.
> > > > >> > >> >
> > > > >> > >> > I intend to start in on hbase2 stability/perf
projects
> after
> > we
> > > > >> > branch.
> > > > >> > >> >
> > > > >> > >> > Interested in any thoughts you all might have
on the above
> > > (Would
> > > > >> also
> > > > >> > >> > appreciate updates on state in [1] if you
are a feature
> > owner).
> > > > >> > >> >
> > > > >> > >> > Thanks,
> > > > >> > >> > St.Ack
> > > > >> > >> >
> > > > >> > >> > 1. https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4
> > > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser
<
> > elserj@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > >
> > > > >> > >> > > Stack wrote:
> > > > >> > >> > >
> > > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh
Elser<
> > > elserj@apache.org>
> > > > >> > >> wrote:
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks for pulling in the FS Quotas
work, Stack. I'm
> > trying
> > > to
> > > > >> > cross
> > > > >> > >> the
> > > > >> > >> > >>> last T's and dot the last I's.
> > > > >> > >> > >>>
> > > > >> > >> > >>> The biggest thing I know I need
to do still is to
> write a
> > > new
> > > > >> > >> chapter
> > > > >> > >> > to
> > > > >> > >> > >>> the book. After that, I'd start
entertaining larger
> > > > >> > >> reviews/discussions
> > > > >> > >> > >>> to
> > > > >> > >> > >>> merge the feature into master.
Anyone with free time
> > > > (giggles)
> > > > >> is
> > > > >> > >> more
> > > > >> > >> > >>> than
> > > > >> > >> > >>> welcome to start perusing :)
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> Out of interest, this could come
in after 2.0 Josh? Any
> > 2.0
> > > > >> > specific
> > > > >> > >> > >> needs
> > > > >> > >> > >> to make this work?
> > > > >> > >> > >>
> > > > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks Josh,
> > > > >> > >> > >> St.Ack
> > > > >> > >> > >>
> > > > >> > >> > >> 1.
> > > > >> > >> > >> https://docs.google.com/document/d/
> > > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> > >>
> > > > >> > >> > >>
> > > > >> > >> > > Nope, no need to block 2.0 on this one
(given the other,
> > > > related
> > > > >> > >> > chatter).
> > > > >> > >> > > Would be nice to get it in, but I completely
understand
> if
> > it
> > > > >> slips
> > > > >> > :)
> > > > >> > >> > >
> > > > >> > >> > > Thanks for updating the doc for me!
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Best regards,
> > > > >> > >>
> > > > >> > >>    - Andy
> > > > >> > >>
> > > > >> > >> If you are given a choice, you believe you have
acted
> freely. -
> > > > >> Raymond
> > > > >> > >> Teller (via Peter Watts)
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message