Just to make sure: 
You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings?

Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.

On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:

We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, writes?

SC

On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net> wrote:
Hi all,

we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)

I found a thread that's about the same problem but on OpenJDK: 

We are on Debian with Sun JDK.

Resident mem is 7.4G while heap is restricted to 3G.

Anyone else is seeing this with Sun JDK?

Cheers,
Daniel

:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

:/home/dd# ps aux |grep java
cass     28201  9.5 46.8 372659544 7707172 ?   SLl  May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                 
28201 cass      20   0  355g 7.4g 1.4g S    8 46.9   5656:25 java