cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Oberman <ober...@civicscience.com>
Subject Re: Survey: Cassandra/JVM Resident Set Size increase
Date Thu, 14 Jul 2011 13:11:19 GMT
I finally upgraded to 0.7.4 -> 0.8.0 (using riptano packages) 2 days ago.
Before, my resident memory (for the java process) would slowly grow without
bound and the OS would kill the process.  But, over the last 2 days, I
_think_ it's been stable.  I'll let you know in a week :-)

My other stats:
AWS large (64 bit, 7.5GB, 4 "compute units", no swap by default and I didn't
enable it manually)
Centos 5.6
Sun  1.6.0_24-b07
2 column families
4 machine cluster with RF=3
Mostly balanced write/read load (usually more writes)
Not quite "big data" volumes, large 10^6 or small 10^7 ops/day
No deletes or mutations, I only add or read

Everything else is stock, I haven't tuned anything as performance was ok.
No JVM options other than what was in the package.  No JNA.  Not sure the GC
patterns.

will

On Tue, Jul 12, 2011 at 9:28 AM, Chris Burroughs
<chris.burroughs@gmail.com>wrote:

> ### Preamble
>
> There have been several reports on the mailing list of the JVM running
> Cassandra using "too much" memory.  That is, the resident set size is
> >>(max java heap size + mmaped segments) and continues to grow until the
> process swaps, kernel oom killer comes along, or performance just
> degrades too far due to the lack of space for the page cache.  It has
> been unclear from these reports if there is a pattern.  My hope here is
> that by comparing JVM versions, OS versions, JVM configuration etc., we
> will find something.  Thank you everyone for your time.
>
>
> Some example reports:
>  - http://www.mail-archive.com/user@cassandra.apache.org/msg09279.html
>  -
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>  - https://issues.apache.org/jira/browse/CASSANDRA-2868
>  -
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/OOM-or-what-settings-to-use-on-AWS-large-td6504060.html
>  -
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-memory-problem-td6545642.html
>
> For reference theories include (in no particular order):
>  - memory fragmentation
>  - JVM bug
>  - OS/glibc bug
>  - direct memory
>  - swap induced fragmentation
>  - some other bad interaction of cassandra/jdk/jvm/os/nio-insanity.
>
> ### Survey
>
> 1. Do you think you are experiencing this problem?
>
> 2.  Why? (This is a good time to share a graph like
> http://www.twitpic.com/5fdabn or
> http://img24.imageshack.us/img24/1754/cassandrarss.png)
>
> 2. Are you using mmap? (If yes be sure to have read
> http://wiki.apache.org/cassandra/FAQ#mmap , and explain how you have
> used pmap [or another tool] to rule you mmap and top decieving you.)
>
> 3. Are you using JNA?  Was mlockall succesful (it's in the logs on
> startup)?
>
> 4. Is swap enabled? Are you swapping?
>
> 5. What version of Apache Cassandra are you using?
>
> 6. What is the earliest version of Apache Cassandra you recall seeing
> this problem with?
>
> 7. Have you tried the patch from CASSANDRA-2654 ?
>
> 8. What jvm and version are you using?
>
> 9. What OS and version are you using?
>
> 10. What are your jvm flags?
>
> 11. Have you tried limiting direct memory (-XX:MaxDirectMemorySize)
>
> 12. Can you characterise how much GC your cluster is doing?
>
> 13. Approximately how many read/writes per unit time is your cluster
> doing (per node or the whole cluster)?
>
> 14.  How are you column families configured (key cache size, row cache
> size, etc.)?
>
>

Mime
View raw message