hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Levin <magn...@gmail.com>
Subject Re: GC and High CPU
Date Wed, 18 May 2011 00:09:53 GMT
Well, our issue was that we take an 8 core box, doing 18MB ParNew GC,
do it super fast (10x1sec), receive RPC calls to RS at the same time,
pile them up due to CPU contention, do GC even faster, run out of user
CPU.  This is basically the cycle of cpu death.

It seems that larger RAM boxes doing this:

      4 2011-05-17T16:52
      4 2011-05-17T16:53
      4 2011-05-17T16:54
      4 2011-05-17T16:55

4 times per minute, is just great for me:

[ParNew: 191744K->21248K(191744K), 0.0361360 secs]

-Jack


-jack

On Tue, May 17, 2011 at 3:39 PM, Todd Lipcon <todd@cloudera.com> wrote:
> My rule of thumb is "as big as possible but no bigger". The young gen size
> determines the pause time and frequency of your young generation
> collections, which are stop-the-world. In my experience the speed is about
> 0.5-1 second per GB on modern hardware.
>
> You get better throughput and less fragmentatoin with a larger young
> genreation, but the pause times go up too.
>
> So, if you have a throughput oriented app, sure, let the JVM use its
> adaptive sizing - otherwise, constrain based on your latency SLAs.
>
> -Todd
>
> On Tue, May 17, 2011 at 7:27 AM, Matt Corgan <mcorgan@hotpads.com> wrote:
>
>> Isn't a 20mb young generation pretty small these days?  For a 4gb heap, you
>> might want to try giving 5-10% of it to the young generation by setting
>>  -XX:NewSize=256M
>>
>> Some of the older
>> docs<
>> http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#1.1.Sizing%20the%20Generations%7Coutline
>> >have
>> a ratio of 1/3 between young/old, so that would be 1gb young, but that
>> is probably bigger than you need.  I'm far from an expert though... what
>> size do other people use?
>>
>> Matt
>>
>>
>> On Tue, May 17, 2011 at 12:55 AM, Jack Levin <magnito@gmail.com> wrote:
>>
>> > This is the way I read it. "Low processors" == "high CPU tasks, e.g. high
>> > load".   So, Incremental takes GC down a number of notches when it comes
>> to
>> > competing with CPU for APP threads.  That being the case the deadlock is
>> > less likely.  It would be useful to add code to the RS that will start
>> > blocking any RPC calls should this condition be detected, if we block say
>> > for 10 seconds, GC could finish doing it 'dirty' work and release CPU :)
>> >
>> >
>> > -Jack
>> >
>> >
>> > On May 16, 2011 5:53 PM, "Stack" <stack@duboce.net> wrote:
>> > > I don't understand what of the below made a difference though the
>> > > difference is plain from the GC logs you show.
>> > >
>> > > See below:
>> > >
>> > > On Mon, May 16, 2011 at 5:06 PM, Jack Levin <magnito@gmail.com> wrote:
>> > >> Those are the lines I added:
>> > >>
>> > >> -XX:+CMSIncrementalMode \  <--------
>> > >
>> > >
>> > > From the doc., it says about i-cms and java6 "This feature is useful
>> > > when applications that need the low pause times provided by the
>> > > concurrent collector are run on machines with small numbers of
>> > > processors (e.g., 1 or 2)." [See
>> > >
>> >
>> http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms
>> > ]
>> > > Don't you have > 2 processors per machine?
>> > >
>> > > St.Ack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message