hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matteo Bertozzi <theo.berto...@gmail.com>
Subject Re: Hourly performance degradation on HBase cluster
Date Fri, 11 Mar 2016 23:28:27 GMT
Master and RegionServer have nothing to do, to each other.
the periodic rolling you see is on the master, and it just remove a file or
two if it needs to.

you are probably hitting GC or compactions on the Region Server which slow
you down in the insert/scan perf.
you can use a monitoring tool to visualize GC, or just add the -X option to
print the GC in the log.
also you can grep for compactions and flushes on the RS logs, to see what
is happening around that time.

Matteo


On Fri, Mar 11, 2016 at 3:25 PM, Vladimir Rodionov <vladrodionov@gmail.com>
wrote:

> HBASE-13202
>
> and Matteo Bertozzi can answer your question?
>
> -Vlad
>
>
>
>
>
> there is
>
>
> On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <gansvv@gmail.com> wrote:
>
> > Hello-
> >
> > I am evaluating HBase for storage of time-series data and see periodic
> > degradation of the cluster happening at the start of the hour, every
> hour,
> > for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in
> the
> > logs during this time like so:
> >
> > 2016-03-01 06:09:35,151 INFO  [WALProcedureStoreSyncThread]
> > wal.WALProcedureStore: Remove log:
> >
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
> >
> > In my tests, I'm pushing 1million data points (each about 200bytes in
> size)
> > into the cluster each minute at the same time using 5000 threads. I am
> also
> > querying 1000 of these data points each minute to test latency, errors,
> and
> > performance. The cluster runs absolutely fine except for those ~10mins at
> > the start of every hour when both reads and writes are affected. Could
> > someone suggest which part of HBase needs tuning? Thanks!
> >
> >
> > -Ganesh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message