Hm - had to digg deeper and it totally
looks like a native mem leak to me:
We are still growing with res += 100MB a day. Cassandra is > 8G
I checked the cassandra process with pmap -x
Here's the human readable (aggregated) output:
Format is thingy: RSS in KB
Total RSS: 8478376
Here's a little more detail:
SSTables (data and index files)
Maybe the output of pmap is totally misleading but my interpretation is
that only 2GB of RSS is attributed to paged in sstables.
I have one large anon block which looks like this:
Address Kbytes RSS
Dirty Mode Mapping
000000073f600000 0 3093248 3093248 rwx--
[ anon ]
This is the native heap thats been allocated on startup and mlocked
So theres still 3.5GB of anon memory.
As I said mem goes up by 100MB each day pretty linearly.
Would be great if anyone could verify this by running pmap or talk my off
the roof by explaining that nothing's the way it seems.
All this might be heavily OS specific so maybe that's only on
Thanks a lot
On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:
mmap'd data will be attributed to res, but the OS can page it
instead of killing the process.
On Mon, Jul 4, 2011 at 5:52
AM, Daniel Doubleday
we have a mem problem with cassandra. res goes up
without bounds (well until
the os kills the process because we dont have
I found a thread that's about the same problem but
We are on Debian with Sun JDK.
Resident mem is 7.4G while heap is restricted to
Anyone else is seeing this with Sun
:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02,
:/home/dd# ps aux |grep java
cass 28201 9.5 46.8 372659544
7707172 ? SLl May24 5656:21
/usr/bin/java -ea -XX:+UseThreadPriorities
-Xms3000M -Xmx3000M -Xmn400M ...
PID USER PR
NI VIRT RES SHR S %CPU %MEM TIME+
28201 cass 20 0
355g 7.4g 1.4g S 8 46.9 5656:25
Project Chair, Apache Cassandra
co-founder of DataStax, the
source for professional Cassandra supporthttp://www.datastax.com