Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE7926075 for ; Mon, 4 Jul 2011 17:19:18 +0000 (UTC) Received: (qmail 22139 invoked by uid 500); 4 Jul 2011 17:19:16 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 21978 invoked by uid 500); 4 Jul 2011 17:19:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 21970 invoked by uid 99); 4 Jul 2011 17:19:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jul 2011 17:19:15 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jul 2011 17:19:10 +0000 Received: by iye7 with SMTP id 7so5695965iye.31 for ; Mon, 04 Jul 2011 10:18:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.115.146 with SMTP id i18mr5843216ibq.22.1309799928596; Mon, 04 Jul 2011 10:18:48 -0700 (PDT) Received: by 10.231.199.18 with HTTP; Mon, 4 Jul 2011 10:18:48 -0700 (PDT) X-Originating-IP: [76.10.173.68] In-Reply-To: References: <75CABA7A-F053-4084-AF9A-101114C72614@gmx.net> Date: Mon, 4 Jul 2011 13:18:48 -0400 Message-ID: Subject: Re: Cassandra memory problem From: Sebastien Coutu To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=005045015a66788a5c04a7418f29 --005045015a66788a5c04a7418f29 Content-Type: text/plain; charset=ISO-8859-1 Hi Daniel, Yes we do see it, since I've added the JNA libraries, it takes a bit more time at that step and locks all the memory. We're using JNA 3.3.0 we've downloaded from there: https://github.com/twall/jna#readme Our servers currently have 32GB of memory and we've assigned 12GB of memory to the Cassandra JVM. We're seeing the following in the logs: INFO [main] 2011-06-27 11:43:14,605 AbstractCassandraDaemon.java (line 97) Heap size: 11811160064/11811160064 INFO [main] 2011-06-27 11:43:21,272 CLibrary.java (line 106) JNA mlockall successful INFO [main] 2011-06-27 11:43:21,292 DatabaseDescriptor.java (line 121) Loading settings from file:/home/hadoop/bin/cassandra/yul01fct/conf/cassandra.yaml INFO [main] 2011-06-27 11:43:21,404 DatabaseDescriptor.java (line 181) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap On the servers, we're seeing a lot of system memory assigned to cache that is "reassigned" to used memory when the applications running on the system really needs it. We're not seeing any swapped memory because we've tweaked swappiness and every application running on the system. We're monitoring the performance of that cluster with Ganglia and see the memory "movement" from the standard graphs produced. Regards, SC On Mon, Jul 4, 2011 at 12:33 PM, Daniel Doubleday wrote: > Hi Sebastian, > > one question: do you use jna.jar and do you see JNA mlockall successful in > your logs. > There's that wild theory here that our problem might be related to mlockall > and no swap. > Maybe the JVM does some realloc stuff and the pinned pages are not cleared > ... > > but that's really only wild guessing. > > Also you are saying that on your servers res mem is not > max heap and the > java process is not swapping? > > Thanks, > Daniel > > On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote: > > It was among one of the issues we had. One of our hosts was using OpenJDK > and we've switched it to Sun and this part of the issue stabilized. The > other issues we had were Heap going through the roof and then OOM under > load. > > > On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday < > daniel.doubleday@gmx.net> wrote: > >> Just to make sure: >> You were seeing that res mem was more than twice of max java heap and that >> did change after you tweaked GC settings? >> >> Note that I am not having a heap / gc problem. The VM itself thinks >> everything is golden. >> >> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote: >> >> We had an issue like that a short while ago here. This was mainly >> happening under heavy load and we managed to stabilize it by tweaking the >> Young/Old space ratio of the JVM and by also tweaking the tenuring >> thresholds/survivor ratios. What kind of load to you have on your systems? >> Mostly reads, writes? >> >> SC >> >> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday < >> daniel.doubleday@gmx.net> wrote: >> >>> Hi all, >>> >>> we have a mem problem with cassandra. res goes up without bounds (well >>> until the os kills the process because we dont have swap) >>> >>> I found a thread that's about the same problem but on OpenJDK: >>> >>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html >>> >>> We are on Debian with Sun JDK. >>> >>> Resident mem is 7.4G while heap is restricted to 3G. >>> >>> Anyone else is seeing this with Sun JDK? >>> >>> Cheers, >>> Daniel >>> >>> :/home/dd# java -version >>> java version "1.6.0_24" >>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >>> >>> :/home/dd# ps aux |grep java >>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 >>> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 >>> -Xms3000M -Xmx3000M -Xmn400M ... >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> >>> >>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java >>> >>> >>> >>> >> >> > > --005045015a66788a5c04a7418f29 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Daniel,

Yes we do see it, since I've added the JN= A libraries, it takes a bit more time at that step and locks all the memory= . We're using JNA 3.3.0 we've downloaded from there:


Our servers currently have 32GB of memory and we&#= 39;ve assigned 12GB of memory to the Cassandra JVM. We're seeing the fo= llowing in the logs:

=A0INFO [main] 2011-06-27 11:43:14,605 AbstractCas= sandraDaemon.java (line 97) Heap size: 11811160064/11811160064
= =A0INFO [main] 2011-06-27 11:43:21,272 CLibrary.java (line 106) JNA mlockal= l successful
=A0INFO [main] 2011-06-27 11:43:21,292 DatabaseDescriptor.java (line 1= 21) Loading settings from file:/home/hadoop/bin/cassandra/yul01fct/conf/cas= sandra.yaml
=A0INFO [main] 2011-06-27 11:43:21,404 DatabaseDescri= ptor.java (line 181) DiskAccessMode 'auto' determined to be mmap, i= ndexAccessMode is mmap

On the servers, we're seeing a lot of system memory= assigned to cache that is "reassigned" to used memory when the a= pplications running on the system really needs it. We're not seeing any= swapped memory because we've tweaked swappiness and every application = running on the system. We're monitoring the performance of that cluster= with Ganglia and see the memory "movement" from the standard gra= phs produced.

Regards,

SC

On Mon, Jul 4, 2011 at 12:33 PM, Daniel Doubleday <daniel.doubleday= @gmx.net> wrote:
Hi= Sebastian,

one question: do you use jna.jar and do you= see=A0JNA mlockall successful in your logs.
There's that wild theory here that our problem might be related to mloc= kall and no swap.=A0
Maybe the JVM does some realloc stuff and th= e pinned pages are not cleared ...=A0

but that'= ;s really only wild guessing.

Also you are saying that on your servers res mem is not= > max heap and the java process is not swapping?

Thanks,
Daniel

On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote:
I= t was among one of the issues we had. One of our hosts was using OpenJDK an= d we've switched it to Sun and this part of the issue stabilized. The o= ther issues we had were Heap going through the roof and then OOM under load= .


On Mon, Jul 4, 2011 at 11:01 AM, Daniel = Doubleday <daniel.doubleday@gmx.net> wrote:
Just to make sure:=A0
You were seei= ng that res mem was more than twice of max java heap and that did change af= ter you tweaked GC settings?

Note that I am not having a= heap / gc problem. The VM itself thinks everything is golden.

On Jul 4, 2011, at 3:41 PM, Sebasti= en Coutu wrote:

We had an issue like tha= t a short while ago here. This was mainly happening under heavy load and we= managed to stabilize it by tweaking the Young/Old space ratio of the JVM a= nd by also tweaking the tenuring thresholds/survivor ratios. What kind of l= oad to you have on your systems? Mostly reads, writes?

SC

On Mon, Jul 4, 201= 1 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net> wrote:
Hi all,

we have a me= m problem with cassandra. res goes up without bounds (well until the os kil= ls the process because we dont have swap)

I found = a thread that's about the same problem but on OpenJDK:=A0

We are on Debian with Sun JDK.

Resident mem is 7.4G while heap is restricted to 3G.

<= div>Anyone else is seeing this with Sun JDK?

Cheers,
Daniel

:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environme= nt (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build = 19.1-b02, mixed mode)

:/home/dd# ps aux |grep java
cass =A0 = =A0 28201 =A09.5 46.8 372659544 7707172 ? =A0 SLl =A0May24 5656:21 /usr/bin= /java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 -Xms3000M = -Xmx3000M -Xmn400M ...

=A0=A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES = =A0SHR S %CPU %MEM =A0 =A0TIME+ =A0COMMAND =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0
28201 cass =A0 =A0 =A020 =A0 0 =A0355g 7.4g 1.4g S =A0 =A08 46.9 =A0 5= 656:25 java



<= /blockquote>



--005045015a66788a5c04a7418f29--