Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 13988 invoked from network); 9 Feb 2011 17:58:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 9 Feb 2011 17:58:03 -0000 Received: (qmail 12528 invoked by uid 500); 9 Feb 2011 17:58:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 12377 invoked by uid 500); 9 Feb 2011 17:57:58 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12369 invoked by uid 99); 9 Feb 2011 17:57:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:57:57 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.213.44] (HELO mail-yw0-f44.google.com) (209.85.213.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:57:49 +0000 Received: by ywk9 with SMTP id 9so217529ywk.31 for ; Wed, 09 Feb 2011 09:57:25 -0800 (PST) MIME-Version: 1.0 Received: by 10.150.52.6 with SMTP id z6mr1963637ybz.340.1297274245401; Wed, 09 Feb 2011 09:57:25 -0800 (PST) Sender: scode@scode.org Received: by 10.151.79.21 with HTTP; Wed, 9 Feb 2011 09:57:25 -0800 (PST) X-Originating-IP: [213.114.156.79] In-Reply-To: References: Date: Wed, 9 Feb 2011 18:57:25 +0100 X-Google-Sender-Auth: 42ME2c3-bR-VprEtvBJkNVzmOAc Message-ID: Subject: Re: Out of control memory consumption From: Peter Schuller To: user@cassandra.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > We are 12-server cluster.=C2=A0 We use random partitioner with manually g= enerated > server tokens.=C2=A0 Memory usage on one server keeps growing out of cont= rol.=C2=A0 We > ran flush and cleared key and row caches but and ran GC but heap memory > usage won't go down.=C2=A0 The only way to heap memory usage to go down i= s the > restart cassandra.=C2=A0 We have to do this one a day.=C2=A0 All other se= rvers have > heap memory usage less than 500MB.=C2=A0 This issue happened on both Cass= andra > 0.6.6 and 0.6.11. To be clear: You are not talking about the size of the Java process in top, but the actual amount of heap used as reported by the JVM via jmx/jconsole/etc? Is the memory amount of memory that you consider high, the heap size just after a concurrent mark/sweep? Are you actually seeing OOM:s or are you restarting the node pre-emptively in response to seeing heap usage go up? > And JVM memory allocation:=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 -Xms3G -Xmx3G Just FYI: So it is entirely expected that the JVM will be 3G (a bit higher) in size (even with standard I/O) and further that the amount of live data in the heap be approaching 3G. The concurrent mark/sweep GC won't trigger until the initial occupancy reaches the limit (if modern Cassandra with default settings). If you've got a 3 gig heap size and the other nodes stay at 500 mb, the question is why *don't* they increase in heap usage. Unless your 500 mb is the report of the actual live data set as evidenced by post-CMS heap usage. --=20 / Peter Schuller