cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rene Kochen <Rene.Koc...@emea.schange.com>
Subject RE: Garbage collection freezes cassandra node
Date Thu, 19 Jan 2012 11:27:29 GMT
Thanks for your comments. The application is indeed suffering from a freezing Cassandra node.
Queries are taking longer than 10 seconds at the moment of a full garbage collect.

Here is an example from the logs. I have a three node cluster. At some point I see on a node
the following log:

21:53:35,986 InetAddress /172.16.107.46 is now dead.

On node "172.16.107.46", I see the following:

21:53:27.192+0100: 1335393.834: [GC 1335393.834: [ParNew (promotion failed): 319468K->324959K(345024K),
0.1304456 secs]1335393.964: [CMS: 6000844K->3298251K(8005248K), 10.8526193 secs] 6310427K->3298251K(8350272K),
[CMS Perm : 26355K->26346K(44268K)], 10.9832679 secs] [Times: user=11.15 sys=0.03, real=10.98
secs] 
21:53:38,174 GC for ConcurrentMarkSweep: 10856 ms for 1 collections, 3389079904 used; max
is 8550678528

I have not yet tested the "XX:+DisableExplicitGC" switch.

Is the right thing to do to decrease the CMSInitiatingOccupancyFraction setting?

Thanks!

Rene

-----Original Message-----
From: scode@scode.org [mailto:scode@scode.org] On Behalf Of Peter Schuller
Sent: dinsdag 20 december 2011 6:38
To: user@cassandra.apache.org
Subject: Re: Garbage collection freezes cassandra node

I should add: If you are indeed actually pausing due to "promotion
failed" or "concurrent mode failure" (which you will see in the GC log
if you enable it with the options I suggested), the first thing I
would try to mitigate is:

* Decrease the occupancy trigger (search for "occupancy") of CMS to a
lower percentage, making the concurrent mark phase start earlier.
* Increase heap size significantly (probably not necessary based on
your graph, but for good measure).

If it then goes away, report back and we can perhaps figure out
details. There are other things that can be done.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Mime
View raw message