hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pun Intended <punintende...@gmail.com>
Subject Re: Very high read rate in a single RegionServer
Date Mon, 26 Jan 2015 02:48:03 GMT
Sorry for the confusion.  I should have used the same place to grab the
numbers from.

In my original email, I grabbed the reqs/sec metric  from the Hbase UI
directly. It was showing 1000 Reqs/sec. Right now, when I look at that, the
number is 891 Reqs/sec, but it fluctuates.

The 55K Reqs/sec in my last email, I got from Cloudera Manager. I am not
sure why it is reporting a different number. It was 55K Reqs/sec, but now
it jumped back up to 60K Reqs/sec.



On Sun, Jan 25, 2015 at 9:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> From your first email:
>
> bq. 1000reqs/sec vs 4-5 reqs/sec elsewhere
>
> Now the request count becomes 55K requests/sec ?
> Just want to clarify whether the request count increased or decreased when
> compaction is not in progress.
>
> Cheers
>
> On Sun, Jan 25, 2015 at 6:23 PM, Pun Intended <punintendedyo@gmail.com>
> wrote:
>
> > Thanks!
> > I ran hbase hbck and all seems healthy.
> >
> > 0 inconsistencies detected.
> > Status: OK
> >
> > Also the meta region shows to be 100% compacted, but I still see 55K
> > requests/sec to that RegionServer:
> >
> >                                            Region
> > Name                                          Num. Compacting KVs Num.
> > Compacted KVs Compaction Progress
> >
> > hbase:meta,,1.1588230740
> > 195349              195349             100.00%
> >
> > best
> >
> >
> > On Sun, Jan 25, 2015 at 8:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > After the adjustment, hbase:meta would still be accessed for region
> > > locations. But the request count should come down noticeably.
> > >
> > > bq. Is there a way to inspect whether I have corruptions in the META
> > table?
> > >
> > > Please take a look at:
> > > http://hbase.apache.org/book.html#hbck.in.depth
> > >
> > > Cheers
> > >
> > > On Sun, Jan 25, 2015 at 5:20 PM, Pun Intended <punintendedyo@gmail.com
> >
> > > wrote:
> > >
> > > > Will do! Thanks you for your suggestion. I will keep an eye on the
> read
> > > > rate to that RegionServer, but in case it doesn't drop, do you think
> > > there
> > > > should be something else I need to look at?
> > > >
> > > > * Is there a way to inspect whether I have corruptions in the META
> > table?
> > > > * Any other possible misconfiguration that comes to mind that I
> should
> > > keep
> > > > an eye for?
> > > >
> > > > best
> > > >
> > > > On Sun, Jan 25, 2015 at 8:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > > >
> > > > > Suggest you lower the frequency of compacting hbase:meta
> > > > >
> > > > > Move the second command to another script.
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > >
> > > > > > On Jan 25, 2015, at 4:46 PM, Pun Intended <
> punintendedyo@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > I have a small script:  major_compact_hbase.hbsh
> > > > > >
> > > > > > major_compact 'users'
> > > > > > major_compact 'hbase:meta'
> > > > > > exit
> > > > > >
> > > > > >
> > > > > > and a cron calls it once a day with:
> > > > > > /usr/bin/hbase shell major_compact_hbase.hbsh
> > > > > >
> > > > > > Thanks again!
> > > > > >
> > > > > >
> > > > > >> On Sun, Jan 25, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> How do you trigger major compaction manually ?
> > > > > >>
> > > > > >> Cheers
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>>> On Jan 25, 2015, at 3:40 PM, Pun Intended <
> > > punintendedyo@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> Thank you so much for your help!
> > > > > >>> You are right, the slow region server hosts hbase:meta!!!
 I am
> > > using
> > > > > >> Hbase
> > > > > >>> 0.96.1.1
> > > > > >>>
> > > > > >>> I have major compactions disabled. I run major compaction
> > manually
> > > > > once a
> > > > > >>> day and when it dpes that besides requesting major compaction
> on
> > my
> > > > > >> tables,
> > > > > >>> it also requests a major_compact 'hbase:meta'. Is that
what
> could
> > > be
> > > > > >>> causing the issue? I did a scan on 'hbase:meta' and
I see that
> it
> > > > > >> contains
> > > > > >>> about the same number of regions as it should - I have
two
> tables
> > > > with
> > > > > >>> ~3000 regions each.
> > > > > >>>
> > > > > >>> Do you think I have some misconfiguration? Do I need
to do some
> > > > cleanup
> > > > > >> on
> > > > > >>> the meta region? Am I hitting some Hbase bug? Or am
I being
> dumb
> > > > > >> requesting
> > > > > >>> major compaction for the meta region daily?
> > > > > >>>
> > > > > >>> I believe on the client side I am doing things right
- using
> > > scanner
> > > > > >>> caching, etc.
> > > > > >>>
> > > > > >>> Thank you so much for your help!!
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message