incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Halpern <e...@dnagamesinc.com>
Subject Re: Nodes dropping out of cluster due to GC
Date Wed, 02 Jun 2010 19:42:27 GMT


Oleg Anastasjev wrote:
> 
>> 
>> Has anyone experienced this sort of problem?  It would be great to hear
>> from
>> anyone who has had experience with this sort of issue and/or suggestions
>> for
>> how to deal with it.
>> 
>> Thanks, Eric  
> 
> Yes, i did. Symptoms you described point to concurrent GC FAILURE. During
> this
> failure concurrent GC completely stops java program (i.e. cassandra) and
> does a
> GC cycle. Other cassandra nodes discover, that node is not responding and
> considering it dead.
> If concurrent GC is properly tuned, it should never do stop-the-world and
> GC ( 
> thats why it is called concurrent ;-) ).
> Reasons for concurrent GC failures can be several:
> 1. Not enought java heap - try to raise max java heap limit
> 2. Improperly sized java heap regions.
> 
> To help you to narrow the problem, pass -XX:+PrintGCDetails option to JVM
> launching cassandra node. This will log information about internal GC
> activities. Let it run till it will be thrown out of cluster again and
> search
> for "concurrent mode failure" or "promotion failed" strings. 
> 

We did indeed have a problem with our GC settings.  The survivor ratio was
too low.  After changing that things are better but we are still seeing GC
that takes 5-10 seconds, which is enough for the node to drop out of the
cluster briefly.
 
-- 
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Nodes-dropping-out-of-cluster-due-to-GC-tp5128481p5132267.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message