Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAFb6HiKh2Y1N10z3a3Z5jfMrRFWKQWZ7zdbu3YGCWrqPF=ca-Q@mail.gmail.com>
References: 
 <CAEaSTZy1+YZ_i36rhfadCwYm+X2Dgz_1EAre1vfOrwAe-VxLpw@mail.gmail.com>
	<CAAg3a2pwcMfwVFr5KvO9QDEGtBww9PUuE8dFpogUuCjEa6EdUA@mail.gmail.com>
	<CAFb6HiKh2Y1N10z3a3Z5jfMrRFWKQWZ7zdbu3YGCWrqPF=ca-Q@mail.gmail.com>
Date: Fri, 11 Mar 2016 16:33:36 -0800
Message-ID: 
 <CAAg3a2qaLgggCrb37Omoid6EgwD8aaNoU+pQK9_iKozQuDT5_A@mail.gmail.com>
Subject: Re: Hourly performance degradation on HBase cluster
From: Vladimir Rodionov <vladrodionov@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a1141228cc61945052dcf32e4

--001a1141228cc61945052dcf32e4
Content-Type: text/plain; charset=UTF-8

The problem is - periodic, with 1h interval, starts at the beginning of
each hour, performance degradation.

I would suspect periodic memstore flusher (flushes memstores every hour, by
default), but it does not explain
why it happens at the beginning of each hour? May be coincidence?

Grep log for "MemstoreFlusherChore".

-Vlad

On Fri, Mar 11, 2016 at 3:28 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
wrote:

> Master and RegionServer have nothing to do, to each other.
> the periodic rolling you see is on the master, and it just remove a file or
> two if it needs to.
>
> you are probably hitting GC or compactions on the Region Server which slow
> you down in the insert/scan perf.
> you can use a monitoring tool to visualize GC, or just add the -X option to
> print the GC in the log.
> also you can grep for compactions and flushes on the RS logs, to see what
> is happening around that time.
>
> Matteo
>
>
> On Fri, Mar 11, 2016 at 3:25 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> wrote:
>
> > HBASE-13202
> >
> > and Matteo Bertozzi can answer your question?
> >
> > -Vlad
> >
> >
> >
> >
> >
> > there is
> >
> >
> > On Fri, Mar 11, 2016 at 3:02 PM, Ganesh V <gansvv@gmail.com> wrote:
> >
> > > Hello-
> > >
> > > I am evaluating HBase for storage of time-series data and see periodic
> > > degradation of the cluster happening at the start of the hour, every
> > hour,
> > > for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in
> > the
> > > logs during this time like so:
> > >
> > > 2016-03-01 06:09:35,151 INFO  [WALProcedureStoreSyncThread]
> > > wal.WALProcedureStore: Remove log:
> > >
> >
> hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log
> > >
> > > In my tests, I'm pushing 1million data points (each about 200bytes in
> > size)
> > > into the cluster each minute at the same time using 5000 threads. I am
> > also
> > > querying 1000 of these data points each minute to test latency, errors,
> > and
> > > performance. The cluster runs absolutely fine except for those ~10mins
> at
> > > the start of every hour when both reads and writes are affected. Could
> > > someone suggest which part of HBase needs tuning? Thanks!
> > >
> > >
> > > -Ganesh
> > >
> >
>

--001a1141228cc61945052dcf32e4--