incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Anastasjev <>
Subject Re: Nodes dropping out of cluster due to GC
Date Wed, 02 Jun 2010 08:14:51 GMT
> Has anyone experienced this sort of problem?  It would be great to hear from
> anyone who has had experience with this sort of issue and/or suggestions for
> how to deal with it.
> Thanks, Eric  

Yes, i did. Symptoms you described point to concurrent GC FAILURE. During this
failure concurrent GC completely stops java program (i.e. cassandra) and does a
GC cycle. Other cassandra nodes discover, that node is not responding and
considering it dead.
If concurrent GC is properly tuned, it should never do stop-the-world and GC ( 
thats why it is called concurrent ;-) ).
Reasons for concurrent GC failures can be several:
1. Not enought java heap - try to raise max java heap limit
2. Improperly sized java heap regions.

To help you to narrow the problem, pass -XX:+PrintGCDetails option to JVM
launching cassandra node. This will log information about internal GC
activities. Let it run till it will be thrown out of cluster again and search
for "concurrent mode failure" or "promotion failed" strings. 

View raw message