hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: CMF & NodeIsDeadException
Date Mon, 03 Jan 2011 22:44:28 GMT
On Mon, Jan 3, 2011 at 2:13 PM, Wayne <wav100@gmail.com> wrote:
> Here are the new settings we are trying out. They seemed to "help" with
> cass. In the end I assume we will need a script to do rolling restarts or
> better yet hbase does it on its own!!
> Thanks for the help!
>        -XX:+UseCMSInitiatingOccupancyOnly
>        -XX:CMSInitiatingOccupancyFraction=60

This seems low.  Means lots of CPU spent GC'ing.  But that said, good
to start low then you can work up from there.

>        -XX:+CMSParallelRemarkEnabled
>        -XX:SurvivorRatio=8
>        -XX:NewRatio=3

This is fine to start with but if it were me, I'd make the young gen
bigger (if objects don't make it up into the tenured heap, they'll not
get in the way of subsequent promotions).  What proportion of heap was
it when you had long pauses?

>        -XX:MaxTenuringThreshold=1

Setting this to 1 means stuff objects get promoted to tenured heap
after surviving only one young GC.  I wonder if you set this to a
higher number how things would run?  (Again, my rationale is that if
objects don't get into the tenured space in the first place, then they
can't be in the way when comes time to promote subsequent objects from
young to tenured.)  It might be something to mess with later.

GC tuning, the "joy of java", is a little bit of a black art.  Its
particularly black given that a bunch think there is no tuning that
will get you away from an occasional stop-the-world GC, at least when
running the CMS collector.

Keep us posted.

> On Mon, Jan 3, 2011 at 5:05 PM, Stack <stack@duboce.net> wrote:
>> On Mon, Jan 3, 2011 at 12:50 PM, Wayne <wav100@gmail.com> wrote:
>> > We have an 8GB heap. What should newsize be? I just had another node die
>> > hard after going into a CMF storm. I swear it had solid CMFs 30+ in a
>> row.
>> >
>> Did a full stop-the-world GC run in between?  It should have cleaned
>> up fragmentation.
>> > I have no idea what eden space is or how to see what it is. ??
>> >
>> Sorry.  There's a bunch of 'cute' terms used for describing the two
>> heap areas in the JVM.  Basically, new stuff goes into the 'new' or
>> 'eden' area first.  If it sticks around through N (configurable) GCs,
>> it gets promoted to old or tenured generation (there are other names
>> for these notions of young and old).  The garbage collection
>> algorithms done in the two heaps differ.  See the Ted citation for
>> more on the gruesome details (though come up to a newer version of
>> that doc).  The JVM is supposed to work ergonomically but it just
>> ain't smart enough dealing w/ HBase/Cass loadings it seems (e.g. it
>> keeps growing the new/eden space pathologically it would seem).
>> > Not knowing what else to do I will start using some of the Cassandra
>> > settings I used to improve it by setting the occupancy fraction. Any
>> other
>> > ideas???
>> >
>> Which config. you talking of? -XX:+CMSInitiatingOccupancyFraction?
>> Thats a good one to toggle down from defaullts.  Should help put off
>> promotion failures a while.
>> St.Ack

View raw message