cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhu Han <>
Subject Re: reduced cached mem; resident set size growth
Date Fri, 28 Jan 2011 09:12:32 GMT

Somebody else and I have the same problem as you, and reported it here:

[NB: It is not solved although the titles said so. Some response from me in
the thread is not accurate.]

IMHO, you should try to upgrade JVM as the first step. If it is solved,
please let us know.

If upgrade of JVM does not work, considering the leak happens on C stack of
JVM,  I suspect it is caused by kernel or glibc, although other java process
on the same box are OK.  But i haven't tries it  as I do not have control of
the kernel version in the VPS environment.

best regards,

On Fri, Jan 28, 2011 at 1:23 AM, Chris Burroughs

> We have a 6 node Cassandra 0.6.8 cluster running on boxes with 4 GB of
> RAM.  Over the course of several weeks cached memory slowly decreases
> until Cassandra is restarted or something bad happens (ie oom killer).
> Performance obviously suffers as cached memory is no longer available.
> Here is a graph of memory over a two week period, the big jump is a
> Cassandra restart:
> There is also tomcat and and your standard linux services running on the
> box as well.   I saved the output of /proc/$CASSANDRA_PID/status ever 10
> seconds from yesterday afternoon to this morning and graphed resident
> set size.  While the jvm and the linux virtual memory system do all
> sorts of clever things, over a period of 12+ hours I believe RSS must go
> in a direction other than up or there is a problem.
> swapiness is currently set to zero and a graph of swap use over a two
> week period does not show any corresponding growth to the lost cached
> space.
> We are using <DiskAccessMode>standard</DiskAccessMode> and no JNA
> interfaces.  I think this makes Cassandra a totally normal java program
> living in heap, so I am confused how this growth could be happening.
> Similar experiences or explanations would be most welcome.  I can
> provide further information if necessary.
> ##### Info dump
> uname: 2.6.18-194.8.1.el5 #1 SMP Wed Jun 23 10:52:51 EDT 2010 x86_64
> x86_64 x86_64 GNU/Linux
> java -version
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> cmd line arg (paths edited):
> /usr/java/jdk1.6.0_20/bin/java -Xms1500M -cXmx1500M -ea -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Dcassandra.compaction.priority=1
> -Dstorage-config=~/conf -Dcassandra-pidfile=~/ -cp
> ~/conf::~/lib/antlr-3.1.3.jar:~/lib/apache-cassandra-0.6.8.jar:~/lib/clhm-production.jar:~/lib/commons-cli-1.1.jar:~/lib/commons-codec-1.2.jar:~/lib/commons-collections-3.2.1.jar:~/lib/commons-lang-2.4.jar:~/lib/google-collections-1.0.jar:~/lib/hadoop-core-0.20.1.jar:~/lib/high-scale-lib.jar:~/lib/ivy-2.1.0.jar:~/lib/jackson-core-asl-1.4.0.jar:~/lib/jackson-mapper-asl-1.4.0.jar:~/lib/jline-0.9.94.jar:~/lib/json-simple-1.1.jar:~/lib/libthrift-r917130.jar:~/lib/log4j-1.2.14.jar:~/lib/slf4j-api-1.5.8.jar:~/lib/slf4j-log4j12-1.5.8.jar:~/lib/zapcass-1.0.0.jar:~/lib/zapcat-1.3-beta.jar
> com.clearspring.cassandra.ZapCassDaemon
> ZapCassDaemon is just getting cassandra and zabbix to talk.  I do not
> believe we are unique in doing this:
> public class ZapCassDaemon {
>    public EmbeddedCassandraService cassandra;
>    public ZabbixAgent agent;
>    public ZapCassDaemon() throws Exception {
>        agent = new ZabbixAgent();
>        cassandra = new EmbeddedCassandraService();
>        cassandra.init();
>    }
>    static public ZapCassDaemon instance;
>    static public void main(String[] args) throws Exception {
>        instance = new ZapCassDaemon();
>    }
> }

View raw message