hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: GC and High CPU
Date Mon, 16 May 2011 02:09:42 GMT
Jack:
Your attachment got filtered.
Can you upload somewhere ?

Thanks

On Sun, May 15, 2011 at 5:37 PM, Jack Levin <magnito@gmail.com> wrote:

> I've added occupancy:  export HBASE_OPTS="$HBASE_OPTS -verbose:gc
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>
> But we had same issue.  Whats strange is that the load on the RS would
> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy,
> and kills all of the user CPU.  I have another jstack:
> http://pastebin.com/4n3MvmrX from a different server.   How do people
> deal with this?  We have HEAP of 12G, block cace at 5G, and upper
> memstore at 1G.  Is hlog roll that intensive?  I though 1000 rpc calls
> per RS should be easy?  The server is not at all busy pre-spike, you
> can tell by attached graph.
>
> -Jack
>
> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <magnito@gmail.com> wrote:
> > Greetings all -- We had some issues with huge GC counts that drive CPU
> > to 100% and rendering RS basically dead, here is some data:
> >
> >  We are normal here, about 1 ParNew per second.
> >
> >    67 2011-05-14T15:35
> >     75 2011-05-14T15:36
> >     70 2011-05-14T15:37
> >     74 2011-05-14T15:38
> >     73 2011-05-14T15:39
> >     73 2011-05-14T15:40
> >     69 2011-05-14T15:41
> >     68 2011-05-14T15:42
> >     76 2011-05-14T15:43
> >     70 2011-05-14T15:44
> >     68 2011-05-14T15:45
> >    322 2011-05-14T15:46
> >    562 2011-05-14T15:47  <------------ This is where its about 10 per
> second.
> >    567 2011-05-14T15:48
> >    563 2011-05-14T15:49
> >    562 2011-05-14T15:50
> >    561 2011-05-14T15:51
> >    513 2011-05-14T15:52
> >
> > here is the jstack from the time of this:
> >
> > http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.
> >
> > This is config for hbase-env:
> >
> > export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> > -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> >
> > Does any have any suggestions on how to manage this?
> >
> > -Jack
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message