From user-return-18418-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Jul 4 16:41:00 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 368556CC9 for ; Mon, 4 Jul 2011 16:41:00 +0000 (UTC) Received: (qmail 65279 invoked by uid 500); 4 Jul 2011 16:40:58 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 65252 invoked by uid 500); 4 Jul 2011 16:40:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 65244 invoked by uid 99); 4 Jul 2011 16:40:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jul 2011 16:40:57 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of daniel.doubleday@gmx.net designates 213.165.64.22 as permitted sender) Received: from [213.165.64.22] (HELO mailout-de.gmx.net) (213.165.64.22) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 04 Jul 2011 16:40:49 +0000 Received: (qmail invoked by alias); 04 Jul 2011 16:33:49 -0000 Received: from p578bde86.dip0.t-ipconnect.de (EHLO caladan.smeet.de) [87.139.222.134] by mail.gmx.net (mp006) with SMTP; 04 Jul 2011 18:33:49 +0200 X-Authenticated: #3445653 X-Provags-ID: V01U2FsdGVkX1/MPqdq8KfMdiWbdpWIVYWla8ACw0UDW22WaY6UUN VBFGIizWrft6Nu From: Daniel Doubleday Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-4-777547451 Subject: Re: Cassandra memory problem Date: Mon, 4 Jul 2011 18:33:49 +0200 In-Reply-To: To: user@cassandra.apache.org References: <75CABA7A-F053-4084-AF9A-101114C72614@gmx.net> Message-Id: X-Mailer: Apple Mail (2.1084) X-Y-GMX-Trusted: 0 X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-4-777547451 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Sebastian, one question: do you use jna.jar and do you see JNA mlockall successful = in your logs. There's that wild theory here that our problem might be related to = mlockall and no swap.=20 Maybe the JVM does some realloc stuff and the pinned pages are not = cleared ...=20 but that's really only wild guessing. Also you are saying that on your servers res mem is not > max heap and = the java process is not swapping? Thanks, Daniel On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote: > It was among one of the issues we had. One of our hosts was using = OpenJDK and we've switched it to Sun and this part of the issue = stabilized. The other issues we had were Heap going through the roof and = then OOM under load. >=20 >=20 > On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday = wrote: > Just to make sure:=20 > You were seeing that res mem was more than twice of max java heap and = that did change after you tweaked GC settings? >=20 > Note that I am not having a heap / gc problem. The VM itself thinks = everything is golden. >=20 > On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote: >=20 >> We had an issue like that a short while ago here. This was mainly = happening under heavy load and we managed to stabilize it by tweaking = the Young/Old space ratio of the JVM and by also tweaking the tenuring = thresholds/survivor ratios. What kind of load to you have on your = systems? Mostly reads, writes? >>=20 >> SC >>=20 >> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday = wrote: >> Hi all, >>=20 >> we have a mem problem with cassandra. res goes up without bounds = (well until the os kills the process because we dont have swap) >>=20 >> I found a thread that's about the same problem but on OpenJDK:=20 >> = http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high= -memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html >>=20 >> We are on Debian with Sun JDK. >>=20 >> Resident mem is 7.4G while heap is restricted to 3G. >>=20 >> Anyone else is seeing this with Sun JDK? >>=20 >> Cheers, >> Daniel >>=20 >> :/home/dd# java -version >> java version "1.6.0_24" >> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >>=20 >> :/home/dd# ps aux |grep java >> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 = /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 = -Xms3000M -Xmx3000M -Xmn400M ... >>=20 >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND = = =20 >> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java >>=20 >>=20 >>=20 >>=20 >=20 >=20 --Apple-Mail-4-777547451 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
It was among one of the issues we had. One of our hosts = was using OpenJDK and we've switched it to Sun and this part of the = issue stabilized. The other issues we had were Heap going through the = roof and then OOM under load.


On Mon, Jul 4, 2011 at 11:01 AM, = Daniel Doubleday <daniel.doubleday@gmx.net><= /span> wrote:
Just to make sure: 
You = were seeing that res mem was more than twice of max java heap and that = did change after you tweaked GC settings?

Note that I = am not having a heap / gc problem. The VM itself thinks everything is = golden.

On Jul 4, 2011, at = 3:41 PM, Sebastien Coutu wrote:

We = had an issue like that a short while ago here. This was mainly happening = under heavy load and we managed to stabilize it by tweaking the = Young/Old space ratio of the JVM and by also tweaking the tenuring = thresholds/survivor ratios. What kind of load to you have on your = systems? Mostly reads, writes?

SC

On Mon, Jul 4, = 2011 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net> wrote:
Hi all,

we have a = mem problem with cassandra. res goes up without bounds (well until the = os kills the process because we dont have = swap)

I found a thread that's about the same = problem but on OpenJDK: 

We are on Debian with Sun = JDK.

Resident mem is 7.4G while heap is = restricted to 3G.

Anyone else is seeing this = with Sun JDK?

Cheers,
Daniel

:/home/dd# java = -version
java version "1.6.0_24"
Java(TM) SE Runtime = Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit = Server VM (build 19.1-b02, mixed mode)

:/home/dd# ps aux |grep java
cass =     28201  9.5 46.8 372659544 7707172 ?   SLl =  May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities = -XX:ThreadPriorityPolicy=3D42 -Xms3000M -Xmx3000M -Xmn400M ...

  PID USER      PR =  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ =  COMMAND                 =                     =                     =                     =                     =                     =                     =         
28201 cass      20   0  355g 7.4g 1.4g S =    8 46.9   5656:25 = java




=


= --Apple-Mail-4-777547451--