Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4916E2001 for ; Thu, 5 May 2011 14:44:42 +0000 (UTC) Received: (qmail 44619 invoked by uid 500); 5 May 2011 14:44:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 44599 invoked by uid 500); 5 May 2011 14:44:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 44580 invoked by uid 99); 5 May 2011 14:44:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 14:44:39 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 14:44:32 +0000 Received: by iyn15 with SMTP id 15so2351645iyn.31 for ; Thu, 05 May 2011 07:44:10 -0700 (PDT) Received: by 10.42.210.70 with SMTP id gj6mr998102icb.236.1304606650616; Thu, 05 May 2011 07:44:10 -0700 (PDT) Received: from mail-iw0-f172.google.com (mail-iw0-f172.google.com [209.85.214.172]) by mx.google.com with ESMTPS id uh10sm846594icb.18.2011.05.05.07.44.10 (version=SSLv3 cipher=OTHER); Thu, 05 May 2011 07:44:10 -0700 (PDT) Received: by iwn39 with SMTP id 39so2347105iwn.31 for ; Thu, 05 May 2011 07:44:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.92.196 with SMTP id s4mr1376431ibm.10.1304606649760; Thu, 05 May 2011 07:44:09 -0700 (PDT) Received: by 10.231.147.2 with HTTP; Thu, 5 May 2011 07:44:09 -0700 (PDT) In-Reply-To: <4DC29CDF.4080400@gmail.com> References: <4DC29CDF.4080400@gmail.com> Date: Thu, 5 May 2011 16:44:09 +0200 Message-ID: Subject: Re: Native heap leaks? From: Thibaut Britz To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec5431e90ee3bf604a288675a X-Virus-Checked: Checked by ClamAV on apache.org --bcaec5431e90ee3bf604a288675a Content-Type: text/plain; charset=ISO-8859-1 Reading this, I tried it again (this time on a freshly formated node due to hd failure). Before crashing, my data dir was only 7.7M big. Using nmap_indexonly (mlockall was successfull) on a 64bit machine. Anything else I could try to get this to work? Ps. All the other nodes (and this node) run fine (even with a few 100 Gigs of data loaded) when I remove the jna.jar, nodetool -h localhost info showing more than 800M of free heap. (of a total of 3256M) Thanks, Thibaut On Thu, May 5, 2011 at 2:49 PM, Chris Burroughs wrote: > On 2011-05-05 06:30, Hannes Schmidt wrote: > > This was my first thought, too. We switched to mmap_index_only and > > didn't see any change in behavior. Looking at the smaps file attached > > to my original post, one can see that the mmapped index files take up > > only a minuscule part of RSS. > > I have not looked into smaps before. But it actually seems odd that that > mmaped Index files are taking up so *little memory*. Are they only a > few kb on disk? Is this a snapshot taken shortly after the process > started or before the OOM killer is presumably about to come along. How > long does it take to go from 1.1 G to 2.1 G resident? Either way, it > would be worthwhile to set one node to standard io to make sure it's > really not mmap causing the problem. > > Anyway, assuming it's not mmap, here are the other similar threads on > the topic. Unfortunately none of them claim an obvious solution: > > http://www.mail-archive.com/user@cassandra.apache.org/msg09279.html > http://www.mail-archive.com/user@cassandra.apache.org/msg08063.html > http://www.mail-archive.com/user@cassandra.apache.org/msg12036.html > http://mail.openjdk.java.net/pipermail/hotspot-dev/2011-April/004091.html > --bcaec5431e90ee3bf604a288675a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Reading this, I tried it again (this time on a freshly formated node due to= hd failure).

Before crashing, my data dir was only 7.7M big. Using= nmap_indexonly (mlockall was successfull) on a 64bit machine.

Anyth= ing else I could try to get this to work? Ps. All the other nodes (and this= node) run fine (even with a few 100 Gigs of data loaded) when I remove the= jna.jar, nodetool=A0 -h localhost info showing more than 800M of free heap= . (of a total of 3256M)

Thanks,
Thibaut








On Thu, May 5, 2011 at 2:49 PM, Chris Burroughs <chris.burroughs@gmail.com= > wrote:
On 2011-05-05 06:30, Hann= es Schmidt wrote:
> This was my first thought, too. We switched to mmap_index_only and
> didn't see any change in behavior. Looking at the smaps file attac= hed
> to my original post, one can see that the mmapped index files take up<= br> > only a minuscule part of RSS.

I have not looked into smaps before. But it actually seems odd that t= hat
mmaped Index files are taking up so *little memory*. =A0Are they only a
few kb on disk? =A0Is this a snapshot taken shortly after the process
started or before the OOM killer is presumably about to come along. =A0How<= br> long does it take to go from 1.1 G to 2.1 G resident? =A0Either way, it
would be worthwhile to set one node to standard io to make sure it's really not mmap causing the problem.

Anyway, assuming it's not mmap, here are the other similar threads on the topic. =A0Unfortunately none of them claim an obvious solution:

http://www.mail-archive.com/user@cassandra.apache.or= g/msg09279.html
http://www.mail-archive.com/user@cassandra.apache.or= g/msg08063.html
http://www.mail-archive.com/user@cassandra.apache.or= g/msg12036.html
http://mail.openjdk.java.net/pipermail/hotspot= -dev/2011-April/004091.html

--bcaec5431e90ee3bf604a288675a--