Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 84B3464B3 for ; Mon, 4 Jul 2011 16:04:51 +0000 (UTC) Received: (qmail 12616 invoked by uid 500); 4 Jul 2011 16:04:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 12522 invoked by uid 500); 4 Jul 2011 16:04:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12514 invoked by uid 99); 4 Jul 2011 16:04:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jul 2011 16:04:48 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jul 2011 16:04:42 +0000 Received: by iwn39 with SMTP id 39so5647838iwn.31 for ; Mon, 04 Jul 2011 09:04:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.115.146 with SMTP id i18mr5787836ibq.22.1309795460813; Mon, 04 Jul 2011 09:04:20 -0700 (PDT) Received: by 10.231.199.18 with HTTP; Mon, 4 Jul 2011 09:04:20 -0700 (PDT) X-Originating-IP: [76.10.173.68] In-Reply-To: References: <75CABA7A-F053-4084-AF9A-101114C72614@gmx.net> Date: Mon, 4 Jul 2011 12:04:20 -0400 Message-ID: Subject: Re: Cassandra memory problem From: Sebastien Coutu To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=005045015a662b953c04a740857d --005045015a662b953c04a740857d Content-Type: text/plain; charset=ISO-8859-1 It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched it to Sun and this part of the issue stabilized. The other issues we had were Heap going through the roof and then OOM under load. On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday wrote: > Just to make sure: > You were seeing that res mem was more than twice of max java heap and that > did change after you tweaked GC settings? > > Note that I am not having a heap / gc problem. The VM itself thinks > everything is golden. > > On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote: > > We had an issue like that a short while ago here. This was mainly happening > under heavy load and we managed to stabilize it by tweaking the Young/Old > space ratio of the JVM and by also tweaking the tenuring thresholds/survivor > ratios. What kind of load to you have on your systems? Mostly reads, writes? > > SC > > On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday > wrote: > >> Hi all, >> >> we have a mem problem with cassandra. res goes up without bounds (well >> until the os kills the process because we dont have swap) >> >> I found a thread that's about the same problem but on OpenJDK: >> >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html >> >> We are on Debian with Sun JDK. >> >> Resident mem is 7.4G while heap is restricted to 3G. >> >> Anyone else is seeing this with Sun JDK? >> >> Cheers, >> Daniel >> >> :/home/dd# java -version >> java version "1.6.0_24" >> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >> >> :/home/dd# ps aux |grep java >> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 >> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 >> -Xms3000M -Xmx3000M -Xmn400M ... >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> >> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java >> >> >> >> > > --005045015a662b953c04a740857d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable It was among one of the issues we had. One of our hosts was using OpenJDK a= nd we've switched it to Sun and this part of the issue stabilized. The = other issues we had were Heap going through the roof and then OOM under loa= d.


On Mon, Jul 4, 2011 at 11:01 AM, Daniel = Doubleday <daniel.doubleday@gmx.net> wrote:
Just to make sure:=A0
You were seei= ng that res mem was more than twice of max java heap and that did change af= ter you tweaked GC settings?

Note that I am not having a= heap / gc problem. The VM itself thinks everything is golden.

On Jul 4, 2011, at 3:4= 1 PM, Sebastien Coutu wrote:

We had an i= ssue like that a short while ago here. This was mainly happening under heav= y load and we managed to stabilize it by tweaking the Young/Old space ratio= of the JVM and by also tweaking the tenuring thresholds/survivor ratios. W= hat kind of load to you have on your systems? Mostly reads, writes?

SC

On Mon, Jul 4, 201= 1 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net> wrote:
Hi all,

we have a me= m problem with cassandra. res goes up without bounds (well until the os kil= ls the process because we dont have swap)

I found = a thread that's about the same problem but on OpenJDK:=A0

We are on Debian with Sun JDK.

Resident mem is 7.4G while heap is restricted to 3G.

<= div>Anyone else is seeing this with Sun JDK?

Cheers,
Daniel

:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environme= nt (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build = 19.1-b02, mixed mode)

:/home/dd# ps aux |grep java
cass =A0 = =A0 28201 =A09.5 46.8 372659544 7707172 ? =A0 SLl =A0May24 5656:21 /usr/bin= /java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 -Xms3000M = -Xmx3000M -Xmn400M ...

=A0=A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES = =A0SHR S %CPU %MEM =A0 =A0TIME+ =A0COMMAND =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0
28201 cass =A0 =A0 =A020 =A0 0 =A0355g 7.4g 1.4g S =A0 =A08 46.9 =A0 5= 656:25 java



<= /blockquote>


--005045015a662b953c04a740857d--