cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Coli <rc...@digg.com>
Subject Re: Cassandra GC Settings
Date Sat, 22 Jan 2011 00:04:59 GMT
On 1/17/11, Dan Hendry <dan.hendry.junk@gmail.com> wrote:
> Since applying these settings, the one time I saw the same type of behavior
> as before, the following appeared in the GC log.
>
>    (concurrent mode failure): 9418431K->6267709K(11841536K), 26.4973750
> secs] 9777393K->6267709K(12290944K), [CMS Perm : 20477K->20443K(34188K)],
> 26.7595510 secs] [Times: user=31.75 sys=0.00, real=26.76 secs]

The symptoms described in both of your mails which mention
pathological cases sound like your heap may simply be too small for
your actual working set. Compaction triggers extra memory pressure and
you then OOM or the concurrent mark sweep fails and you thrash.

It is also worth noting that  major compaction reduces the
effectiveness of various caches (internal and o/s level), and it is
somewhat likely that your node has internally backed up threadpools
during and immediately after compaction. Some work has been done
recently to improve these characteristics, but I don't think those
changes are in 0.7.0 release.

> INFO [ScheduledTasks:1] 2011-01-17 10:42:15,911 GCInspector.java (line 133) GC for ConcurrentMarkSweep:
45828 ms, 3350764696 reclaimed leaving 9224048472 used; max is 12783583232

As the amount of headroom available to the CMS GCer decreases, it
tends to take longer and longer to reclaim less and less memory. 458
seconds to recover 3gb (leaving 9gb heap out of a max heap of 16gb?)
suggests that your working set has put you into this grey area where
it eventually works but sucks really badly. This is just before the
state where it permanently locks up the JVM and/or OOMs.

Have you sized your memtables and caches so that you have meaningful
heap headroom when your caches are full?

=Rob

Mime
View raw message