hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Very high read rate in a single RegionServer
Date Mon, 26 Jan 2015 02:37:51 GMT
>From your first email:

bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere

Now the request count becomes 55K requests/sec ?
Just want to clarify whether the request count increased or decreased when
compaction is not in progress.

Cheers

On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended <punintendedyo@gmail.com>
wrote:

> Thanks!
> I ran hbase hbck and all seems healthy.
>
> 0 inconsistencies detected.
> Status: OK
>
> Also the meta region shows to be 100% compacted, but I still see 55K
> requests/sec to that RegionServer:
>
>                                            Region
> Name                                          Num. Compacting KVs Num.
> Compacted KVs Compaction Progress
>
> hbase:meta,,1.1588230740
> 195349              195349             100.00%
>
> best
>
>
> On Sun, Jan 25, 2015 at 8:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > After the adjustment, hbase:meta would still be accessed for region
> > locations. But the request count should come down noticeably.
> >
> > bq. Is there a way to inspect whether I have corruptions in the META
> table?
> >
> > Please take a look at:
> > http://hbase.apache.org/book.html#hbck.in.depth
> >
> > Cheers
> >
> > On Sun, Jan 25, 2015 at 5:20 PM, Pun Intended <punintendedyo@gmail.com>
> > wrote:
> >
> > > Will do! Thanks you for your suggestion. I will keep an eye on the read
> > > rate to that RegionServer, but in case it doesn't drop, do you think
> > there
> > > should be something else I need to look at?
> > >
> > > * Is there a way to inspect whether I have corruptions in the META
> table?
> > > * Any other possible misconfiguration that comes to mind that I should
> > keep
> > > an eye for?
> > >
> > > best
> > >
> > > On Sun, Jan 25, 2015 at 8:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > Suggest you lower the frequency of compacting hbase:meta
> > > >
> > > > Move the second command to another script.
> > > >
> > > > Cheers
> > > >
> > > >
> > > >
> > > > > On Jan 25, 2015, at 4:46 PM, Pun Intended <punintendedyo@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hi Ted,
> > > > >
> > > > > I have a small script:  major_compact_hbase.hbsh
> > > > >
> > > > > major_compact 'users'
> > > > > major_compact 'hbase:meta'
> > > > > exit
> > > > >
> > > > >
> > > > > and a cron calls it once a day with:
> > > > > /usr/bin/hbase shell major_compact_hbase.hbsh
> > > > >
> > > > > Thanks again!
> > > > >
> > > > >
> > > > >> On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > >>
> > > > >> How do you trigger major compaction manually ?
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >>
> > > > >>
> > > > >>>> On Jan 25, 2015, at 3:40 PM, Pun Intended <
> > punintendedyo@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> Thank you so much for your help!
> > > > >>> You are right, the slow region server hosts hbase:meta!!!
 I am
> > using
> > > > >> Hbase
> > > > >>> 0.96.1.1
> > > > >>>
> > > > >>> I have major compactions disabled. I run major compaction
> manually
> > > > once a
> > > > >>> day and when it dpes that besides requesting major compaction
on
> my
> > > > >> tables,
> > > > >>> it also requests a major_compact 'hbase:meta'. Is that what
could
> > be
> > > > >>> causing the issue? I did a scan on 'hbase:meta' and I see
that it
> > > > >> contains
> > > > >>> about the same number of regions as it should - I have two
tables
> > > with
> > > > >>> ~3000 regions each.
> > > > >>>
> > > > >>> Do you think I have some misconfiguration? Do I need to do
some
> > > cleanup
> > > > >> on
> > > > >>> the meta region? Am I hitting some Hbase bug? Or am I being
dumb
> > > > >> requesting
> > > > >>> major compaction for the meta region daily?
> > > > >>>
> > > > >>> I believe on the client side I am doing things right - using
> > scanner
> > > > >>> caching, etc.
> > > > >>>
> > > > >>> Thank you so much for your help!!
> > > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message