Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 13715 invoked from network); 21 Apr 2010 12:47:10 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 Apr 2010 12:47:10 -0000 Received: (qmail 1842 invoked by uid 500); 21 Apr 2010 12:47:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 1801 invoked by uid 500); 21 Apr 2010 12:47:09 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 1793 invoked by uid 99); 21 Apr 2010 12:47:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 12:47:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nithril@gmail.com designates 209.85.218.222 as permitted sender) Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 12:47:00 +0000 Received: by bwz22 with SMTP id 22so7029725bwz.5 for ; Wed, 21 Apr 2010 05:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type; bh=Y5nnYjETLnnjdwo53L4I0z0SXBVCtRh4wPXJZ3ZLEEE=; b=WescEyPKEFxlCh9Cto5HOrdFQ/4na8J6KLq5IheqqpLZQdCXF3rGb4P/A0Myov9+Uv i1oSQ3olye00waEXtuWwLKtoiEsRguqlWbaErVYx84cZ7BIqRVytM9zqxQELfeAPflef hCwbJZk4n0K8BqFD2aVYLdcbqAZVG/hpnGqrQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=NpuBL0aM1XGfqUFNvHxXlAr3YLE1XQRs31B4gVxSOHsXrvyUaXnnE6vWiyxw9ZyNuP KAy28H+OqkK0XfvoiwVnZGAzdK1YFz1763KaM8MCv0pt34RRZhAHdV+fiOwbIlenP0WB iCoDFpJ7Xnd2bAvgoX8nPyJE3N3gslgglcvS8= MIME-Version: 1.0 Received: by 10.204.99.143 with HTTP; Wed, 21 Apr 2010 05:46:39 -0700 (PDT) In-Reply-To: References: Date: Wed, 21 Apr 2010 14:46:39 +0200 Received: by 10.204.141.78 with SMTP id l14mr2758662bku.85.1271853999948; Wed, 21 Apr 2010 05:46:39 -0700 (PDT) Message-ID: Subject: Re: Cassandra tuning for running test on a desktop From: Nicolas Labrot To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001517478d96df85e00484be95df X-Virus-Checked: Checked by ClamAV on apache.org --001517478d96df85e00484be95df Content-Type: text/plain; charset=ISO-8859-1 I have try 1400M, and Cassandra OOM too. Is there another solution ? My data isn't very big. It seems that is the merge of the db On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene wrote: > Trying increasing Xmx. 1G is probably not enough for the amount of inserts > you are doing. > > > On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot wrote: > >> Hello, >> >> For my first message I will first thanks Cassandra contributors for their >> great works. >> >> I have a parameter issue with Cassandra (I hope it's just a parameter >> issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's a >> simple dual core with 4GB of RAM on WinXP. I have keep the default JVM >> option inside cassandra.bat (Xmx1G) >> >> I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF >> (named Super1). The insertion go to 1 millions of SC (without slowdown) and >> Cassandra crash because of an OOM. (I store an average of 100 bytes per SC >> with a max of 10kB). >> I have aggressively decreased all the memories parameters without any >> respect to the consistency (My config is here [1]), the cache is turn off >> but Cassandra still go to OOM. I have joined the last line of the Cassandra >> life [2]. >> >> What can I do to fix my issue ? Is there another solution than increasing >> the Xmx ? >> >> Thanks for your help, >> >> Nicolas >> >> >> >> >> >> [1] >> >> >> > ColumnType="Super" >> CompareWith="BytesType" >> CompareSubcolumnsWith="BytesType" /> >> >> org.apache.cassandra.locator.RackUnawareStrategy >> 1 >> >> org.apache.cassandra.locator.EndPointSnitch >> >> >> 32 >> >> auto >> 64 >> 64 >> 16 >> 4 >> 64 >> >> 16 >> 32 >> 0.01 >> 0.01 >> 60 >> 4 >> 8 >> >> >> >> [2] >> INFO 13:36:41,062 Super1 has reached its threshold; switching in a fresh >> Memtable at >> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log', >> position=5417524) >> INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755 >> INFO 13:36:41,062 Writing Memtable(Super1)@15385755 >> INFO 13:36:42,062 Completed flushing >> d:\cassandra\data\Keyspace1\Super1-711-Data.db >> INFO 13:36:45,781 Super1 has reached its threshold; switching in a fresh >> Memtable at >> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log', >> position=6065637) >> INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910 >> INFO 13:36:45,796 Writing Memtable(Super1)@15578910 >> INFO 13:36:46,109 Completed flushing >> d:\cassandra\data\Keyspace1\Super1-712-Data.db >> INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240 reclaimed >> leaving 922392600 used; max is 1174208512 >> INFO 13:36:54,593 Super1 has reached its threshold; switching in a fresh >> Memtable at >> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log', >> position=6722241) >> INFO 13:36:54,593 Enqueuing flush of Memtable(Super1)@24468872 >> INFO 13:36:54,593 Writing Memtable(Super1)@24468872 >> INFO 13:36:55,421 Completed flushing >> d:\cassandra\data\Keyspace1\Super1-713-Data.dbjava.lang.OutOfMemoryError: >> Java heap space >> INFO 13:37:08,281 GC for ConcurrentMarkSweep: 5561 ms, 9432 reclaimed >> leaving 971904520 used; max is 1174208512 >> >> > --001517478d96df85e00484be95df Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have try 1400M, and Cassandra OOM too.

Is there another solution ?= My data isn't very big.

It seems that is the merge of the db

On Wed, Apr 21, 2010 at 2:14 PM, Mark G= reene <greenemj@= gmail.com> wrote:
Trying increasing= Xmx. 1G is probably not enough for the amount of inserts you are doing.


On Wed, Apr= 21, 2010 at 8:10 AM, Nicolas Labrot <nithril@gmail.com> wro= te:
Hello,

For= my first message I will first thanks Cassandra contributors for their grea= t works.

I have a parameter issue with Cassandra (I hope it's just a paramet= er issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. = It's a simple dual core with 4GB of RAM on WinXP. I have keep the defau= lt JVM option inside cassandra.bat (Xmx1G)

I'm trying to insert 3 millions of SC with 6 Columns each inside 1 = CF (named Super1). The insertion go to 1 millions of SC (without slowdown) = and Cassandra crash because of an OOM. (I store an average of 100 bytes per= SC with a max of 10kB).
I have aggressively decreased all the memories parameters without any respe= ct to the consistency (My config is here [1]), the cache is turn off but Ca= ssandra still go to OOM. I have joined the last line of the Cassandra life = [2].

What can I do to fix my issue ?=A0 Is there another solution than incre= asing the Xmx ?

Thanks for your help,

Nicolas



=

[1]
=A0 <Keyspaces>
=A0=A0=A0 <Keyspace Name=3D"= ;Keyspace1">
=A0=A0=A0=A0=A0 <ColumnFamily Name=3D"Super1"
=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ColumnType=3D"Super"= ;
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 CompareWith= =3D"BytesType"
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0 CompareSubcolumnsWith=3D"BytesType" />
=A0=A0=A0=A0=A0 <ReplicaPlacementStrategy>org.apache.cassandra.locato= r.RackUnawareStrategy</ReplicaPlacementStrategy>
=A0=A0=A0=A0=A0 &= lt;ReplicationFactor>1</ReplicationFactor>
=A0=A0=A0=A0=A0 <= EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointS= nitch>
=A0=A0=A0 </Keyspace>
=A0 </Keyspaces>
=A0 <CommitLogR= otationThresholdInMB>32</CommitLogRotationThresholdInMB>

= =A0 <DiskAccessMode>auto</DiskAccessMode>
=A0 <RowWarning= ThresholdInMB>64</RowWarningThresholdInMB>
=A0 <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
=A0 <= FlushDataBufferSizeInMB>16</FlushDataBufferSizeInMB>
=A0 <Fl= ushIndexBufferSizeInMB>4</FlushIndexBufferSizeInMB>
=A0 <Col= umnIndexSizeInKB>64</ColumnIndexSizeInKB>

=A0 <MemtableThroughputInMB>16</MemtableThroughputInMB>
= =A0 <BinaryMemtableThroughputInMB>32</BinaryMemtableThroughputInMB= >
=A0 <MemtableOperationsInMillions>0.01</MemtableOperations= InMillions>
=A0 <MemtableObjectCountInMillions>0.01</MemtableObjectCountInMill= ions>
=A0 <MemtableFlushAfterMinutes>60</MemtableFlushAfterM= inutes>
=A0 <ConcurrentReads>4</ConcurrentReads>
=A0 &= lt;ConcurrentWrites>8</ConcurrentWrites>
</Storage>


[2]
=A0INFO 13:36:41,062 Super1 has reached = its threshold; switching in a fresh Memtable at CommitLogContext(file=3D= 9;d:/cassandra/commitlog\CommitLog-1271849783703.log', position=3D54175= 24)
=A0INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
=A0INF= O 13:36:41,062 Writing Memtable(Super1)@15385755
=A0INFO 13:36:42,062 Co= mpleted flushing d:\cassandra\data\Keyspace1\Super1-711-Data.db
=A0INFO = 13:36:45,781 Super1 has reached its threshold; switching in a fresh Memtabl= e at CommitLogContext(file=3D'd:/cassandra/commitlog\CommitLog-12718497= 83703.log', position=3D6065637)
=A0INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
=A0INF= O 13:36:45,796 Writing Memtable(Super1)@15578910
=A0INFO 13:36:46,109 Co= mpleted flushing d:\cassandra\data\Keyspace1\Super1-712-Data.db
=A0INFO = 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240 reclaimed leavin= g 922392600 used; max is 1174208512
=A0INFO 13:36:54,593 Super1 has reached its threshold; switching in a fresh= Memtable at CommitLogContext(file=3D'd:/cassandra/commitlog\CommitLog-= 1271849783703.log', position=3D6722241)
=A0INFO 13:36:54,593 Enqueui= ng flush of Memtable(Super1)@24468872
=A0INFO 13:36:54,593 Writing Memtable(Super1)@24468872
=A0INFO 13:36:55,= 421 Completed flushing d:\cassandra\data\Keyspace1\Super1-713-Data.dbjava.l= ang.OutOfMemoryError: Java heap space
=A0INFO 13:37:08,281 GC for Concur= rentMarkSweep: 5561 ms, 9432 reclaimed leaving 971904520 used; max is 11742= 08512



--001517478d96df85e00484be95df--