Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4C2264CFA for ; Thu, 7 Jul 2011 12:20:18 +0000 (UTC) Received: (qmail 83544 invoked by uid 500); 7 Jul 2011 12:20:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 83379 invoked by uid 500); 7 Jul 2011 12:20:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 83371 invoked by uid 99); 7 Jul 2011 12:20:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2011 12:20:14 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.214.44 is neither permitted nor denied by domain of oberman@civicscience.com) Received: from [209.85.214.44] (HELO mail-bw0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2011 12:20:09 +0000 Received: by bwb17 with SMTP id 17so948436bwb.31 for ; Thu, 07 Jul 2011 05:19:47 -0700 (PDT) Received: by 10.204.70.76 with SMTP id c12mr679180bkj.70.1310041187068; Thu, 07 Jul 2011 05:19:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.81.36 with HTTP; Thu, 7 Jul 2011 05:19:26 -0700 (PDT) X-Originating-IP: [24.131.19.240] In-Reply-To: References: <75CABA7A-F053-4084-AF9A-101114C72614@gmx.net> From: William Oberman Date: Thu, 7 Jul 2011 08:19:26 -0400 Message-ID: Subject: Re: Cassandra memory problem To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636eeedcc98a9b004a779bb3d --001636eeedcc98a9b004a779bb3d Content-Type: text/plain; charset=ISO-8859-1 I think I had (and have) a similar problem: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/OOM-or-what-settings-to-use-on-AWS-large-td6504060.html My memory usage grew slowly until I ran out of mem and the OS killed my process (due to no swap). I'm still on 0.7.4, but I'm rolling out 0.8.1 next week, which I was hoping would fix the problem. I'm using Centos with Sun 1.6.0_24-b07 will On Thu, Jul 7, 2011 at 7:41 AM, Daniel Doubleday wrote: > Hm - had to digg deeper and it totally looks like a native mem leak to me: > > We are still growing with res += 100MB a day. Cassandra is > 8G now > > I checked the cassandra process with pmap -x > > Here's the human readable (aggregated) output: > > Format is thingy: RSS in KB > > Summary: > > Total SST: 1961616 > Anon RSS: 6499640 > > Total RSS: 8478376 > > Here's a little more detail: > > SSTables (data and index files) > ****** > Attic: 0 > PrivateChatNotification: 38108 > Schema: 0 > PrivateChat: 161048 > UserData: 116788 > HintsColumnFamily: 0 > Rooms: 100548 > Tracker: 476 > Migrations: 0 > ObjectRepository: 793680 > BlobStore: 350924 > Activities: 400044 > LocationInfo: 0 > > Libraries > ****** > javajar: 2292 > nativelib: 13028 > > Other > ****** > 28201: 32 > jna979649866618987247.tmp: 92 > locale-archive: 1492 > [stack]: 132 > java: 44 > ffi8TsQPY(deleted): 8 > > And > ****** > [anon]: 6499640 > > > Maybe the output of pmap is totally misleading but my interpretation is > that only 2GB of RSS is attributed to paged in sstables. > I have one large anon block which looks like this: > > Address Kbytes RSS Dirty Mode Mapping > 000000073f600000 0 3093248 3093248 rwx-- [ anon ] > > This is the native heap thats been allocated on startup and mlocked > > So theres still 3.5GB of anon memory. > > We haven't deployed https://issues.apache.org/jira/browse/CASSANDRA-2654 yet > and this might be part of it but I don't think thats the main problem. > As I said mem goes up by 100MB each day pretty linearly. > > Would be great if anyone could verify this by running pmap or talk my off > the roof by explaining that nothing's the way it seems. > > All this might be heavily OS specific so maybe that's only on Debian? > > Thanks a lot > Daniel > > On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote: > > mmap'd data will be attributed to res, but the OS can page it out > instead of killing the process. > > On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday > wrote: > > Hi all, > > we have a mem problem with cassandra. res goes up without bounds (well > until > > the os kills the process because we dont have swap) > > I found a thread that's about the same problem but on OpenJDK: > > > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html > > We are on Debian with Sun JDK. > > Resident mem is 7.4G while heap is restricted to 3G. > > Anyone else is seeing this with Sun JDK? > > Cheers, > > Daniel > > :/home/dd# java -version > > java version "1.6.0_24" > > Java(TM) SE Runtime Environment (build 1.6.0_24-b07) > > Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) > > :/home/dd# ps aux |grep java > > cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 > > /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 > > -Xms3000M -Xmx3000M -Xmn400M ... > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > > > --001636eeedcc98a9b004a779bb3d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I think I had (and have) a similar problem:
My memory usage grew slowly until I ran out of mem and the OS killed m= y process (due to no swap).

I'm still on 0.7.4= , but I'm rolling out 0.8.1 next week, which I was hoping would fix the= problem. =A0I'm using Centos with Sun=A01.6.0_24-b07

will

On Thu, Jul 7, 2= 011 at 7:41 AM, Daniel Doubleday <daniel.doubleday@gmx.net><= /span> wrote:
Hm - had to digg deeper and it totally = looks like a native mem leak to me:

We are still growing= with res +=3D 100MB a day. Cassandra is > 8G now

I checked the cassandra process with pmap -x

Here&= #39;s the human readable (aggregated) output:

Form= at is=A0thingy: RSS in KB

Summary:

Total SST: 1961616
Anon RSS:=A064= 99640

Total RSS: 8478376

Here's a little more detail:

SSTables (data and index files)
******
Attic:= 0
PrivateChatNotification: 38108
Schem= a: 0
PrivateChat: 161048
UserData: 116788
HintsColumnFamily: 0
Rooms: 100548
Tracker: 476
Migrations: 0
ObjectRepository: 793680
BlobStore: 350924
Activities:= 400044
LocationInfo: 0

Libraries
***= ***
javajar: 2292
nativelib: 13028

=
Other
******
28201: 32
jna97964986661898= 7247.tmp: 92
locale-archive: 1492
[stack]: 132
java: 44
ffi8TsQPY(deleted): 8

And
******
[anon]: 6499640


Maybe the output of pmap is totally misleading b= ut my interpretation is that only 2GB of RSS is attributed to paged in ssta= bles.
I have one large=A0anon=A0block which looks like this:

Address =A0 =A0 =A0 =A0 =A0 Kbytes =A0 =A0 RSS =A0= Dirty Mode =A0 Mapping
000000073f600000 =A0 =A0 =A0 0 3093248 30= 93248 rwx-- =A0 =A0[ anon ]

This is the nati= ve heap thats been allocated on startup and mlocked

So theres still 3.5GB of anon memory.

We haven't deployed=A0https://issues.apache.org/jira/br= owse/CASSANDRA-2654=A0yet and this might be part of it but I don't = think thats the main problem.
As I said mem goes up by 100MB each day pretty linearly.
Would be great if anyone could verify this by running pmap or t= alk my off the roof by explaining that nothing's the way it seems.

All this might be heavily OS specific so maybe that'= ;s only on Debian?

Thanks a lot
Daniel= =A0

On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:

=
mmap'd data will be attr= ibuted to res, but the OS can page it out
instead of killing the process= .

On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
<daniel.doubleday@gmx.net= > wrote:
Hi all,
we have a mem problem with cassandra. res goes up without bounds (well unti= l
the os kills the process becaus= e we dont have swap)
I found a th= read that's about the same problem but on OpenJDK:
http://cassandra= -user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilizati= on-not-caused-by-mmap-on-sstables-td5840777.html
We are on Debian with Sun JDK.
Resident mem is 7.4G while heap is res= tricted to 3G.
Anyone else is see= ing this with Sun JDK?
Cheers,
Daniel
:/home/dd# j= ava -version
java version "1= .6.0_24"
Java(TM) SE Runtime Environment (bui= ld 1.6.0_24-b07)
Java HotSpot(TM)= 64-Bit Server VM (build 19.1-b02, mixed mode)
:/home/dd# ps aux |grep java
cass= =A0 =A0 28201 =A09.5 46.8 372659544 7707172 ? =A0 SLl =A0May24 5656:21
=
/usr/bin/java -ea -XX:+UseThreadPrio= rities -XX:ThreadPriorityPolicy=3D42
-Xms3000M -Xmx3000M -Xmn400M ...
=
=A0=A0PID USER =A0 =A0 =A0PR =A0NI = =A0VIRT =A0RES =A0SHR S %CPU %MEM =A0 =A0TIME+ =A0COMMAND
<= blockquote type=3D"cite">

28201 cass =A0 =A0 =A020 =A0 0 =A0355g 7.4g 1.4g S =A0 =A08 46.9 =A0= 5656:25 java






-- =
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Data= Stax, the source for professional Cassandra support
http://www.datastax.com



--001636eeedcc98a9b004a779bb3d--