From user-return-10679-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Nov 22 19:00:52 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 10390 invoked from network); 22 Nov 2010 19:00:52 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 22 Nov 2010 19:00:52 -0000 Received: (qmail 61298 invoked by uid 500); 22 Nov 2010 19:01:19 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 61281 invoked by uid 500); 22 Nov 2010 19:01:19 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61268 invoked by uid 99); 22 Nov 2010 19:01:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 19:01:19 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a48.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 19:01:09 +0000 Received: from homiemail-a48.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTP id 07CBB4F806B for ; Mon, 22 Nov 2010 11:00:35 -0800 (PST) Received: from localhost (webms.mac.com [17.148.16.116]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTPSA id 3E7704F8076 for ; Mon, 22 Nov 2010 11:00:19 -0800 (PST) To: user@cassandra.apache.org From: Aaron Morton Subject: Re: Newbie question on Cassandra mem usage Date: Mon, 22 Nov 2010 19:00:18 GMT X-Mailer: MobileMe Mail (1C3207) Message-id: <3923633e-aee2-6586-b880-b6862ce5c4d1@me.com> Content-Type: multipart/alternative; boundary=Apple-Webmail-42--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd MIME-Version: 1.0 In-Reply-To: X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Webmail-42--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; format=flowed The higher memory usage for the java process may be because of memory mapp= ed file access, take a look at the disk_access_mode in cassandra.yaml.=A0=0A= =0AWRT going OutOfMemory:=0A- what are your Memtable thresholds in cassand= ra.yaml ?=A0=0A- how many Column Families do you have?=A0=0A- What are you= r row and key cache settings?=0A- Have a read of JVM HeapSize section here= =A0http://wiki.apache.org/cassandra/MemtableThresholds=0A- Have a read of=A0= http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts=0A=0AI= n short, if you've turned up any memory settings turn them down. Run your = test again and see if it completes. Then turn them up a little at a time.=A0= =0A=0AIf you're still having trouble include some details of your cassandr= a.yaml file and the schema definition next time. As well as how many cassa= ndra nodes you have, how many clients you are running against it and how f= ast they are writing.=A0=0A=0AAaron=0A=0A=0AOn 23 Nov, 2010,at 07:45 AM, T= rung Tran wrote:=0A=0AHi,=0A=0AI have a test cl= uster of 3 nodes, 14Gb of mem in each node,=0Areplication factor =3D 3. Wi= th default -Xms and Xmx, my nodes are set to=0Ahave max-heap-size =3D 7Gb.= After initial load with about 200M rows=0A(write with hector default cons= istencylevel =3D quorum,) my nodes memory=0Ausage are up to 13.5Gb, show a= bunches of GC notifications and=0Aeventually crashes with java.lang.OutOf= MemoryError: Java heap space.=0A=0AIs there any setting that can help with= this scenario?=0A=0AThanks,=0ATrung.=0A --Apple-Webmail-42--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd Content-Type: multipart/related; type="text/html"; boundary=Apple-Webmail-86--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd --Apple-Webmail-86--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1;
The higher memory usage for the java process may be because of memory= mapped file access, take a look at the disk_access_mode in cassandra.yaml=  

WRT going OutOfMemory:
- what a= re your Memtable thresholds in cassandra.yaml ? 
- how many= Column Families do you have? 
- What are your row and key = cache settings?
- Have a read of JVM HeapSize section here = http://wik= i.apache.org/cassandra/MemtableThresholds

In short, if you've turned up a= ny memory settings turn them down. Run your test again and see if it compl= etes. Then turn them up a little at a time. 

If you're still having trouble include some details of your cassandra.yam= l file and the schema definition next time. As well as how many cassandra = nodes you have, how many clients you are running against it and how fast t= hey are writing. 

Aaron

=

On 23 Nov, 2010,at 07:45 AM, Trung Tran <tran.hieutr= ung@gmail.com> wrote:

=
Hi,
=0A
=0AI have a test cluster of 3 nodes,= 14Gb of mem in each node,
=0Areplication factor =3D 3. With default -X= ms and Xmx, my nodes are set to
=0Ahave max-heap-size =3D 7Gb. After in= itial load with about 200M rows
=0A(write with hector default consisten= cylevel =3D quorum,) my nodes memory
=0Ausage are up to 13.5Gb, show a = bunches of GC notifications and
=0Aeventually crashes with java.lang.Ou= tOfMemoryError: Java heap space.
=0A
=0AIs there any setting that ca= n help with this scenario?
=0A
=0AThanks,
=0ATrung.
=0A
<= /div>
--Apple-Webmail-86--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd-- --Apple-Webmail-42--9cd4cc35-f077-6d1a-e7e9-ccaaed75dacd--