incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: Out of control memory consumption
Date Wed, 09 Feb 2011 17:57:25 GMT
> We are 12-server cluster.  We use random partitioner with manually generated
> server tokens.  Memory usage on one server keeps growing out of control.  We
> ran flush and cleared key and row caches but and ran GC but heap memory
> usage won't go down.  The only way to heap memory usage to go down is the
> restart cassandra.  We have to do this one a day.  All other servers have
> heap memory usage less than 500MB.  This issue happened on both Cassandra
> 0.6.6 and 0.6.11.

To be clear: You are not talking about the size of the Java process in
top, but the actual amount of heap used as reported by the JVM via

Is the memory amount of memory that you consider high, the heap size
just after a concurrent mark/sweep?

Are you actually seeing OOM:s or are you restarting the node
pre-emptively in response to seeing heap usage go up?

> And JVM memory allocation:          -Xms3G -Xmx3G

Just FYI: So it is entirely expected that the JVM will be 3G (a bit
higher) in size (even with standard I/O) and further that the amount
of live data in the heap be approaching 3G. The concurrent mark/sweep
GC won't trigger until the initial occupancy reaches the limit (if
modern Cassandra with default settings).

If you've got a 3 gig heap size and the other nodes stay at 500 mb,
the question is why *don't* they increase in heap usage. Unless your
500 mb is the report of the actual live data set as evidenced by
post-CMS heap usage.

/ Peter Schuller

View raw message