Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates
 74.125.83.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=IY7hW2tKBIYRj0I0gbsUpuIjtrBAaWMBTWwKkEiwJzu+Kd70b3EIg/5CCg5d6UWUft
         G5a6FQ8fNl4teHDhQv8uBezjBXrGaTuZSGCDgivas8f15pwNKuHhe+p8u6zORUG+WseA
         q0FJo5wjE4uVDBIyMaJD/7pIOfmbQckJ3Rhyo=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <BANLkTimgqBqPU7nnvh=isd6qXAxug7xBVg@mail.gmail.com>
References: <BANLkTinwfro6cQNoAyn3dQ+wJb6UtZ8HUA@mail.gmail.com>
	<BANLkTimgqBqPU7nnvh=isd6qXAxug7xBVg@mail.gmail.com>
Date: Mon, 16 May 2011 11:53:31 -0700
Message-ID: <BANLkTinfJBiA8VghHk98vupHn4uzdYp6fw@mail.gmail.com>
Subject: Re: GC and High CPU
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you have a high insert rate then maybe log rolling (which blocks
inserts a little) makes it that the calls get queued enough (occupying
heap) to make you enter a GC loop of death? Can you enable RPC logging
and see if you can confirm that?

Thx,

J-D

On Sun, May 15, 2011 at 5:37 PM, Jack Levin <magnito@gmail.com> wrote:
> I've added occupancy: =A0export HBASE_OPTS=3D"$HBASE_OPTS -verbose:gc
> -XX:CMSInitiatingOccupancyFraction=3D70 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>
> But we had same issue. =A0Whats strange is that the load on the RS would
> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
> and kills all of the user CPU. =A0I have another jstack:
> http://pastebin.com/4n3MvmrX from a different server. =A0 How do people
> deal with this? =A0We have HEAP of 12G, block cace at 5G, and upper
> memstore at 1G. =A0Is hlog roll that intensive? =A0I though 1000 rpc call=
s
> per RS should be easy? =A0The server is not at all busy pre-spike, you
> can tell by attached graph.
>
> -Jack
>
> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <magnito@gmail.com> wrote:
>> Greetings all -- We had some issues with huge GC counts that drive CPU
>> to 100% and rendering RS basically dead, here is some data:
>>
>> =A0We are normal here, about 1 ParNew per second.
>>
>> =A0 =A067 2011-05-14T15:35
>> =A0 =A0 75 2011-05-14T15:36
>> =A0 =A0 70 2011-05-14T15:37
>> =A0 =A0 74 2011-05-14T15:38
>> =A0 =A0 73 2011-05-14T15:39
>> =A0 =A0 73 2011-05-14T15:40
>> =A0 =A0 69 2011-05-14T15:41
>> =A0 =A0 68 2011-05-14T15:42
>> =A0 =A0 76 2011-05-14T15:43
>> =A0 =A0 70 2011-05-14T15:44
>> =A0 =A0 68 2011-05-14T15:45
>> =A0 =A0322 2011-05-14T15:46
>> =A0 =A0562 2011-05-14T15:47 =A0<------------ This is where its about 10 =
per second.
>> =A0 =A0567 2011-05-14T15:48
>> =A0 =A0563 2011-05-14T15:49
>> =A0 =A0562 2011-05-14T15:50
>> =A0 =A0561 2011-05-14T15:51
>> =A0 =A0513 2011-05-14T15:52
>>
>> here is the jstack from the time of this:
>>
>> http://pastebin.com/RYdwH8Gp =A0Unusually high BLOCKED threads.
>>
>> This is config for hbase-env:
>>
>> export HBASE_OPTS=3D"$HBASE_OPTS -verbose:gc =A0-XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>> Does any have any suggestions on how to manage this?
>>
>> -Jack
>>
>