hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Date Tue, 07 Sep 2010 04:32:37 GMT
On Mon, Sep 6, 2010 at 9:19 PM, Stack <stack@duboce.net> wrote:

> Nice one Todd. Its a little like HBASE-2880.  The issue you found
> looks like it has a few guises.  I also think it an issue we've always
> had, not a regression.
>

I don't think so. I killed the server that hit the issue, and within a
couple minutes it happened on another server. I've never seen this before
and this is the same sort of test that I've run on every other release.

Looking at the diff of HBASE-2461, I think the difference is that we used to
close the region and then do the writes into META for the new info. Since
the region was already marked closed by the time we were writing, we
wouldn't have handlers stuck on the lock. Post-HBASE-2461, the META writes
happen under the lock before the region gets marked closed.


>
> Things are different since new master commit last week -- there are
> multiple executors per operation type, open, close, etc., with root
> and meta having their own instantiations -- but you need a handler
> before you get an executor.   The case where all handlers could end up
> occupied, in this case 'blocked' on a particular region lock, and no
> progress can be made because an edit needs to be made back into the
> same server before we can progress looks like it can still happen,
> even in the new regime.
>
> I agree we should just keep moving forward.  Lets fix in the next 0.89.x.
>

Are other people not seeing this issue under a load test? I got it within an
hour using YCSB with 1G split sizes - I imagine if you tune it down to a
stress-test config with lots of splits, you'll see it even faster.

-Todd


> On Mon, Sep 6, 2010 at 6:20 PM, Todd Lipcon <todd@cloudera.com> wrote:
> > I did some load tests on this afternoon and ran into this bug:
> >
> > https://issues.apache.org/jira/browse/HBASE-2964
> >
> > I got this after loading only 44GB (with 1G region size), so I don't
> think
> > it's that rare. I'm also running with something like 30 handlers. My
> tests
> > are with 80 concurrent clients.
> >
> > So consider me a -0 - if no one else runs into this bug we may as well
> > release, but it seems a step backward in stability for me from the
> 20100621
> > (the last one I did significant load testing on)
> >
> > -Todd
> >
> > On Fri, Sep 3, 2010 at 12:05 PM, Stack <stack@duboce.net> wrote:
> >
> >> +1
> >>
> >> Its running in production at SU.
> >>
> >> St.Ack
> >>
> >> On Mon, Aug 30, 2010 at 5:29 PM, Jean-Daniel Cryans <
> jdcryans@apache.org>
> >> wrote:
> >> > BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
> >> > deployed it in almost all our environments here (this will be
> >> > completed before the voting period ends).
> >> >
> >> > J-D
> >> >
> >> > On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <
> jdcryans@apache.org>
> >> wrote:
> >> >> It's time for another DR since the new master code is about to be
> >> >> merged in and we have a few fixes and improvements that'd beneficiate
> >> >> from more exposure. I branched from trunk this morning and created
a
> >> >> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
> >> >> more on what these 0.89.x releases are all about)
> >> >>
> >> >> Source binary and source tar balls are available here:
> >> >>
> >> >>  http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
> >> >>
> >> >> You can also browse the candidate documentation here:
> >> >>
> >> >>
> >>
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
> >> >>
> >> >> Issues resolved since 0.89.20100726, our second 0.89.x release, are
> >> >> roughly ~23 issues odd including fixed deadlocks, better handling of
> >> >> IOEs during splits and improvements for filters: see
> >> >> http://su.pr/2HwiUe
> >> >>
> >> >> Shall we release this candidate as the third in our 0.89.x series of
> >> >> developer releases?
> >> >>
> >> >> Please see previous threads on 0.89 releases for more information
> >> >> about the purpose of this release candidate - in particular, this
> >> >> 'developer release' is for those who can tolerate risk and who are
> >> >> willing to give feedback in advance of our next major release.  We're
> >> >> not making any guarantees that this is bug free. Its definitely not
> >> >> for production deploys.
> >> >>
> >> >> We'll do another release like this in a few weeks after the new
> master
> >> >> code has gone in.
> >> >>
> >> >> Please vote by Monday, September 6th.
> >> >>
> >> >> Thanks,
> >> >> J-D
> >> >>
> >> >
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message