hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Re: Need Help: RegionTooBusyException: Above memstore limit
Date Wed, 04 Mar 2015 04:21:56 GMT
The error disappeared after changing write buffer from 20MB to 2MB. Thanks
for the help!

Jianshi

On Wed, Mar 4, 2015 at 12:12 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> It depends on how you manage your connection, your table and your puts. If
> it works for you with reducing the batch buffer size, then just keep it the
> way it is...
>
> JM
>
> 2015-03-03 11:10 GMT-05:00 Jianshi Huang <jianshi.huang@gmail.com>:
>
> > Yes, looks like reducing the batch buffer size works (still validating).
> >
> > But why setAutoFlush(false) is harmful here? I just want maximum write
> > speed.
> >
> > Jianshi
> >
> > On Tue, Mar 3, 2015 at 10:54 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Let HBase manage the flushes for you. Remove
> > edgeTable.setAutoFlush(false)
> > > and maybe reduce your batch size.
> > >
> > > I don't think that increasing the memstore is the good way to go. Sound
> > > more like a plaster on the issue than a good fix (for me).
> > >
> > > JM
> > >
> > > 2015-03-03 9:43 GMT-05:00 Ted Yu <yuzhihong@gmail.com>:
> > >
> > > > Default value for hbase.regionserver.global.memstore.size is 0.4
> > > >
> > > > Meaning Maximum size of all memstores in the region server before new
> > > > updates
> > > > are blocked and flushes are forced is 7352m which is lower than 774m.
> > > >
> > > > You can increase the value for
> hbase.regionserver.global.memstore.size
> > > >
> > > > Please also see if you can distribute the writes to the underlying
> > region
> > > > so that the region's use of memstore comes down.
> > > >
> > > > Cheersx
> > > >
> > > > On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > Only one region server is problematic.
> > > > >
> > > > > hbase.regionserver.global.memstore.size is not set, the problematic
> > > > region
> > > > > is using 774m for memstore.
> > > > >
> > > > > Max heap is 18380m for all region servers.
> > > > >
> > > > > Jianshi
> > > > >
> > > > >
> > > > > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >
> > > > > > What's the value for hbase.regionserver.global.memstore.size
?
> > > > > >
> > > > > > Did RegionTooBusyException happen to many regions or only a
few
> > > > regions ?
> > > > > >
> > > > > > How much heap did you give region servers ?
> > > > > >
> > > > > > bq. HBase version is 0.98.0.2.1.2.0-402
> > > > > >
> > > > > > Yeah, this is a bit old. Please consider upgrading.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <
> > > jianshi.huang@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm constantly facing "RegionTooBusyException: Above memstore
> > > limit"
> > > > > > errors
> > > > > > > in one region server when writing data to HBase.
> > > > > > >
> > > > > > > I checked the region server log, and I've seen a lot of
> warnings
> > > > during
> > > > > > the
> > > > > > > data writes:
> > > > > > >
> > > > > > >   WARN wal.fshlog couldn't find oldest seqNum for the region
> > we're
> > > > > about
> > > > > > to
> > > > > > > flush, ...
> > > > > > >
> > > > > > > Then HBase seem to flush the data and added it as a HStore
> file.
> > > > > > >
> > > > > > > I also get a few warnings in client.ShortCircuitCache,
says
> > "could
> > > > not
> > > > > > load
> > > > > > > ... due to InvalidToken exceptions.
> > > > > > >
> > > > > > > Anyone can give me hint what went wrong?
> > > > > > >
> > > > > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1,
but
> > the
> > > > > > release
> > > > > > > is a little bit old.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Jianshi Huang
> > > > > > >
> > > > > > > LinkedIn: jianshi
> > > > > > > Twitter: @jshuang
> > > > > > > Github & Blog: http://huangjs.github.com/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Jianshi Huang
> > > > >
> > > > > LinkedIn: jianshi
> > > > > Twitter: @jshuang
> > > > > Github & Blog: http://huangjs.github.com/
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>



-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message