cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Ricardo Motta Gomes <paulo.mo...@chaordicsystems.com>
Subject Nodes get stuck in crazy GC loop after some time, leading to timeouts
Date Fri, 28 Nov 2014 20:55:59 GMT
Hello,

This is a recurrent behavior of JVM GC in Cassandra that I never completely
understood: when a node is UP for many days (or even months), or receives a
very high load spike (3x-5x normal load), CMS GC pauses start becoming very
frequent and slow, causing periodic timeouts in Cassandra. Trying to run GC
manually doesn't free up memory. The only solution when a node reaches this
state is to restart the node.

We restart the whole cluster every 1 or 2 months, to avoid machines getting
into this crazy state. We tried tuning GC size and parameters, different
cassandra versions (1.1, 1.2, 2.0), but this behavior keeps happening. More
recently, during black friday, we received about 5x our normal load, and
some machines started presenting this behavior. Once again, we restart the
nodes an the GC behaves normal again.

I'm attaching a few pictures comparing the heap of "healthy" and "sick"
nodes: http://imgur.com/a/Tcr3w

You can clearly notice some memory is actually reclaimed during GC in
healthy nodes, while in sick machines very little memory is reclaimed.
Also, since GC is executed more frequently in sick machines, it uses about
2x more CPU than non-sick nodes.

Have you ever observed this behavior in your cluster? Could this be related
to heap fragmentation? Would using the G1 collector help in this case? Any
GC tuning or monitoring advice to troubleshoot this issue?

Any advice or pointers will be kindly appreciated.

Cheers,

-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200

Mime
View raw message