It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched it to Sun and this part of the issue stabilized. The other issues we had were Heap going through the roof and then OOM under load.
Just to make sure:You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings?Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, writes?SCOn Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <firstname.lastname@example.org> wrote:
Hi all,we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)I found a thread that's about the same problem but on OpenJDK:http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.htmlWe are on Debian with Sun JDK.Resident mem is 7.4G while heap is restricted to 3G.Anyone else is seeing this with Sun JDK?Cheers,Daniel:/home/dd# java -versionjava version "1.6.0_24"Java(TM) SE Runtime Environment (build 1.6.0_24-b07)Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode):/home/dd# ps aux |grep javacass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java