cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Nodes frozen in GC
Date Tue, 08 Mar 2011 12:20:59 GMT
> So, are you saying this is normal and expected from Cassandra?  So,
> under load, we can expect java garbage collection to stop the Cassandra
> process on that server from time to time, essentially taking out the
> node for short periods of time while it does garbage collection?

This thread is getting out of hand and out into la-la-land. Original
poster: If you want to skip some rantings of mine, skip to the end and
do (1) and get the results to the list.

First of all, re-checking the history, it seems the only concrete
information is:

INFO [ScheduledTasks:1] 2011-03-05 15:21:23,524 (line
128) GC for ConcurrentMarkSweep: 18052 ms, -997761672 reclaimed
leaving 5796586088

This indicates that a concurrent mark/sweep GC took 18 seconds. That
may or may not be a bit high for the heap size, but regardless, the
CMS is not a stop-the-world pause. It involves some stop-the-world
pauses, but those 18 seconds aren't it.

I still don't see anything that tells what's actually going on in the
OP's case. But the fact that the heap grew rather than shrunk as a
result of the GC cycle suggests that something is indeed wrong.
Probably the heap is too full, as has already been suggested, and the
question is just why. Probably *something* is tweaked incorrectly for
the heap size, be that row cache, memtable flush thresholds, etc. Or
there's a bug. But there seems to be a distinct lack of information
and a distinct non-lack of random speculating and GC blaming.

CASSANDRA-2252 as was linked to is *not* a magic fix for this.

A lot can be said about garbage collection techniques, but the point
is that the point of the CMS collector is to avoid the need for long
stop-the-world pauses. Some are still required, but they are supposed
to normally be short. For some workloads, you eventually reach a point
where fragmentation in the old generation causes the need do to a full
stop-the-world pause while the entire heap is compacted. This *does*
result in a long uninterrupted pause, if/when it happens.

Usually, it happens because you actually have too much live data on
the heap. That is entirely different from having a reasonable workload
that is still not handled by the GC in a sensible fashion.

Is it possible that everything is configured correctly and the OP is
triggering a bug or just triggering a sufficiently poor behavior of
CMS such that the freezes are due to unavoidable periodic compactions?
Yes. Do we have nearly enough information to know? No. Should we
assume it is the GC/JVM:s fault before having such information, given
that lots of people run Cassandra without triggering this to the
extent this seems to imply? No.

I would suggest to the OP:

(1) I cannot stress this one enough: Run with -XX:+PrintGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps and collect the output.
(2) Attach to your process with jconsole or some similar tool.
(3) Observe the behavior of the heap over time. Preferably post
screenshots so others can look at them.

(1) in particular is very important. It's completely useless to be
speculating about details and making sweeping statements when all
indications so far indicate that there is too much live data on the
heap, when there is not even the results of (1) to go by.

(1) will give you output which shows when different stages of GC
trigger and information about heap sizes etc. It will also print the
reason for fallback to full GC, such as a promotion failure. One can
usually observe fairly well what lead up to such a fallback and draw
conclusions. It will also show which stages took what amount of time
(and not all of them are stop-the-world).

/ Peter Schuller

View raw message