hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: GC and High CPU
Date Mon, 16 May 2011 18:53:31 GMT
If you have a high insert rate then maybe log rolling (which blocks
inserts a little) makes it that the calls get queued enough (occupying
heap) to make you enter a GC loop of death? Can you enable RPC logging
and see if you can confirm that?

Thx,

J-D

On Sun, May 15, 2011 at 5:37 PM, Jack Levin <magnito@gmail.com> wrote:
> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>
> But we had same issue.  Whats strange is that the load on the RS would
> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
> and kills all of the user CPU.  I have another jstack:
> http://pastebin.com/4n3MvmrX from a different server.   How do people
> deal with this?  We have HEAP of 12G, block cace at 5G, and upper
> memstore at 1G.  Is hlog roll that intensive?  I though 1000 rpc calls
> per RS should be easy?  The server is not at all busy pre-spike, you
> can tell by attached graph.
>
> -Jack
>
> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <magnito@gmail.com> wrote:
>> Greetings all -- We had some issues with huge GC counts that drive CPU
>> to 100% and rendering RS basically dead, here is some data:
>>
>>  We are normal here, about 1 ParNew per second.
>>
>>    67 2011-05-14T15:35
>>     75 2011-05-14T15:36
>>     70 2011-05-14T15:37
>>     74 2011-05-14T15:38
>>     73 2011-05-14T15:39
>>     73 2011-05-14T15:40
>>     69 2011-05-14T15:41
>>     68 2011-05-14T15:42
>>     76 2011-05-14T15:43
>>     70 2011-05-14T15:44
>>     68 2011-05-14T15:45
>>    322 2011-05-14T15:46
>>    562 2011-05-14T15:47  <------------ This is where its about 10 per second.
>>    567 2011-05-14T15:48
>>    563 2011-05-14T15:49
>>    562 2011-05-14T15:50
>>    561 2011-05-14T15:51
>>    513 2011-05-14T15:52
>>
>> here is the jstack from the time of this:
>>
>> http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.
>>
>> This is config for hbase-env:
>>
>> export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>>
>> Does any have any suggestions on how to manage this?
>>
>> -Jack
>>
>

Mime
View raw message