cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Watson <>
Subject Long GC due to promotion failures
Date Tue, 21 Jan 2014 08:02:22 GMT
Pretty reliable, at some point, nodes will have super long GCs.
Followed by

Lovely log messages:

  9030.798: [ParNew (0: promotion failure size = 4194306)  (2:
promotion failure size = 4194306)  (4: promotion failure size =
4194306)  (promotion failed)
  Total time for which application threads were stopped: 23.2659990 seconds

Full gc.log until just before restarting the node (see another 32s GC
near the end):

Here's graph of GC time, where we can see a an increase 30 minutes
prior (indicator that the issue will happen soon):

Graph of various Heap usage:

Running compactions in the same time frame:

CPU, IO, ops and latencies:


Cassandra 1.2.13
Oracle JDK 1.6u45

JVM opts:


Tried HEAP_NEW_SIZE of 768M, 800M, 1000M and 1600M
Tried default "-XX:SurvivorRatio=8" and "-XX:SurvivorRatio=4"
Tried default "-XX:MaxTenuringThreshold=1" and "-XX:MaxTenuringThreshold=2"

All still eventually ran into long GC.

Hardware for all 3 nodes:

(2) E5520 @ 2.27Ghz (8 cores w/ HT) ["16" cores]
(6) 4GB RAM [24G RAM]
(1) 500GB 7.2k for commitlog
(2) 400G SSD for data (configured as separate data directories)

View raw message