hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Sechrist <ssechr...@gmail.com>
Subject Re: Garbage collection issues
Date Wed, 24 Nov 2010 20:52:35 GMT
Interesting. The settings we tried earlier today slowed jobs significantly,
but no failures (yet). We're going to try the 512MB newSize and 60%
CMSInitiatingOccupancyFraction. 1 second pauses here and there would be OK
for us.... we just want to avoid the long pauses right now. We'll also do
what we can to avoid swapping. The ganglia metrics on on there.

Thanks,
Sean

On Wed, Nov 24, 2010 at 3:34 PM, Todd Lipcon <todd@cloudera.com> wrote:

> On Wed, Nov 24, 2010 at 7:01 AM, Sean Sechrist <ssechrist@gmail.com>wrote:
>
>> Hey guys,
>>
>> I just want to get an idea about how everyone avoids these long GC pauses
>> that cause regionservers to die.
>>
>> What kind of java heap and garbage collection settings do you use?
>>
>> What do you do to make sure that the HBase vm never uses swap? I have
>> heard
>> turning off swap altogether can be dangerous, so right now we have the
>> setting vm.swappiness=0. How do you tell if it's using swap? On Ganglia,
>> we
>> see the "CPU wio" metric at around 4.5% before one of our crashes. Is that
>> high?
>>
>> To try to avoid using too much memory, is reducing the memstore
>> upper/lower
>> limit, or the block cache size a good idea? Should we just tune down
>> HBase's
>> total heap to try to avoid swap?
>>
>> In terms of our specific problem:
>>
>> We seem to keep running into garbage collection pauses that cause the
>> regionservers to die. We have mix of some random read jobs, as well as a
>> few
>> full-scan jobs (~1.5 billion rows, 800-900GB of data, 1500 regions), and
>> we
>> are always inserting data. We would rather sacrifice a little speed for
>> stability, if that means anything. We have 7 nodes (RS + DN + TT) with
>> 12GB
>> max heap given to HBase, and 24GB memory total.
>>
>> We were using the following garbage collection options:
>> -XX:+UseConcMarkSweepGC -XX:NewSize=64m -XX:MaxNewSize=64m
>> -XX:CMSInitiatingOccupancyFraction=75
>>
>> After looking at http://wiki.apache.org/hadoop/PerformanceTuning, we are
>> trying to lower NewSize/MaxNewSize to 6m as well as reducing
>> CMSInitiatingOccupancyFraction to 50.
>>
>
> Rather than reducing the new size, you should consider increasing new size
> if you're OK with higher latency but fewer long GC pauses.
>
> GC is a complicated subject, but here are a few rules of thumb:
>
> - A larger young generation means that the young GC pauses, which are
> stop-the-world, will take longer. In my experience it's somewhere around 1
> second per GB of new size. So, if you're OK with periodic 1 second pauses, a
> large (1GB) new size should be fine.
> - A larger young generation also means that less data will get tenured to
> the old generation. This means that the old generation will have to collect
> less often and also that it will become less fragmented.
> - In HBase, the long (45second+) pauses generally happen when promotion
> fails due to heap fragmentation in the old generation. So, it falls back to
> stop-the-world compacting collection which takes a long time.
>
> So, in general, a large young gen will reduce the frequency of super-long
> pauses, but will increase the frequency of shorter pauses.
>
> It sounds like you may be OK with longer young gen pauses, so maybe
> consider new size at 512M with your 12G total heap?
>
> I also wouldn't tune CMSInitiatingOccupancy below 60% - that will cause CMS
> to always be running which isn't that efficient.
>
> -Todd
>
>
>>
>> We see messages like this in our GC logs:
>>
>>> 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs]
>>
>>   (concurrent mode failure): 10126729K->5760080K(13246464K), 91.2530340
>>> secs]
>>
>>
>>
>> 2010-11-23T14:56:01.383-0500: 61297.449: [GC 61297.449: [ParNew (promotion
>> failed): 57425K->57880K(59008K), 0.1880950 secs]61297.637:
>> [CMS2010-11-23T14:56:06.336-0500: 61302.402: [CMS-concurrent-mark:
>> 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs]
>>  (concurrent mode failure): 10126729K->5760080K(13246464K), 91.2530340
>> secs]
>> 10181961K->5760080K(13305472K), [CMS Perm : 20252K->20241K(33868K)],
>> 91.4413320 secs] [Times: user=24.47 sys=1.07, real=91.44 secs]
>>
>> There's a lot of questions there, but I definitely appreciate any advice
>> or
>> input anybody else has. Thanks so much!
>>
>> -Sean
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message