hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: Hourly performance degradation on HBase cluster
Date Sat, 12 Mar 2016 00:33:36 GMT
The problem is - periodic, with 1h interval, starts at the beginning of
each hour, performance degradation.

I would suspect periodic memstore flusher (flushes memstores every hour, by
default), but it does not explain
why it happens at the beginning of each hour? May be coincidence?

Grep log for "MemstoreFlusherChore".

-Vlad

On Fri, Mar 11, 2016 at 3:28 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
wrote:

> Master and RegionServer have nothing to do, to each other.
> the periodic rolling you see is on the master, and it just remove a file or
> two if it needs to.
>
> you are probably hitting GC or compactions on the Region Server which slow
> you down in the insert/scan perf.
> you can use a monitoring tool to visualize GC, or just add the -X option to
> print the GC in the log.
> also you can grep for compactions and flushes on the RS logs, to see what
> is happening around that time.
>
> Matteo
>
>
> On Fri, Mar 11, 2016 at 3:25 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> wrote:
>
> > HBASE-13202
> >
> > and Matteo Bertozzi can answer your question?
> >
> > -Vlad
> >
> >
> >
> >
> >
> > there is
> >
> >
> > On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <gansvv@gmail.com> wrote:
> >
> > > Hello-
> > >
> > > I am evaluating HBase for storage of time-series data and see periodic
> > > degradation of the cluster happening at the start of the hour, every
> > hour,
> > > for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in
> > the
> > > logs during this time like so:
> > >
> > > 2016-03-01 06:09:35,151 INFO  [WALProcedureStoreSyncThread]
> > > wal.WALProcedureStore: Remove log:
> > >
> >
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
> > >
> > > In my tests, I'm pushing 1million data points (each about 200bytes in
> > size)
> > > into the cluster each minute at the same time using 5000 threads. I am
> > also
> > > querying 1000 of these data points each minute to test latency, errors,
> > and
> > > performance. The cluster runs absolutely fine except for those ~10mins
> at
> > > the start of every hour when both reads and writes are affected. Could
> > > someone suggest which part of HBase needs tuning? Thanks!
> > >
> > >
> > > -Ganesh
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message