incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Out of control memory consumption
Date Wed, 09 Feb 2011 17:57:25 GMT
> We are 12-server cluster.  We use random partitioner with manually generated
> server tokens.  Memory usage on one server keeps growing out of control.  We
> ran flush and cleared key and row caches but and ran GC but heap memory
> usage won't go down.  The only way to heap memory usage to go down is the
> restart cassandra.  We have to do this one a day.  All other servers have
> heap memory usage less than 500MB.  This issue happened on both Cassandra
> 0.6.6 and 0.6.11.

To be clear: You are not talking about the size of the Java process in
top, but the actual amount of heap used as reported by the JVM via
jmx/jconsole/etc?

Is the memory amount of memory that you consider high, the heap size
just after a concurrent mark/sweep?

Are you actually seeing OOM:s or are you restarting the node
pre-emptively in response to seeing heap usage go up?


> And JVM memory allocation:          -Xms3G -Xmx3G

Just FYI: So it is entirely expected that the JVM will be 3G (a bit
higher) in size (even with standard I/O) and further that the amount
of live data in the heap be approaching 3G. The concurrent mark/sweep
GC won't trigger until the initial occupancy reaches the limit (if
modern Cassandra with default settings).

If you've got a 3 gig heap size and the other nodes stay at 500 mb,
the question is why *don't* they increase in heap usage. Unless your
500 mb is the report of the actual live data set as evidenced by
post-CMS heap usage.

-- 
/ Peter Schuller

Mime
View raw message