Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 18971 invoked from network); 6 Dec 2010 20:35:20 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Dec 2010 20:35:20 -0000 Received: (qmail 13931 invoked by uid 500); 6 Dec 2010 20:35:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13834 invoked by uid 500); 6 Dec 2010 20:35:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13826 invoked by uid 99); 6 Dec 2010 20:35:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 20:35:18 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a40.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 20:35:11 +0000 Received: from homiemail-a40.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a40.g.dreamhost.com (Postfix) with ESMTP id 9271174C081 for ; Mon, 6 Dec 2010 12:34:44 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=to:from :subject:date:message-id:content-type:mime-version:in-reply-to; q=dns; s=thelastpickle.com; b=JGU980hI+UrbLhWbqw/ufvdmxch53IHom T3yVjtaWS0rpzzANdJQLVmCs5uXZwyyQXmPYpikDRCBpDrl3MZH0SXME1h1sg9ef RGe5li8e3m5Mn+6f5BACY9QmYKUJ+Ar/9WRwzWZnXsFZBYABI3nk8cVhcH6tfeFP LcZ2vjrVyI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=to :from:subject:date:message-id:content-type:mime-version: in-reply-to; s=thelastpickle.com; bh=bYNK5XHMus3n2XS2PLKb0bO3UuE =; b=ulXtLOfd/wXnspNVTtTkoE0ufcbGc8HQkpvrF6ce4lBRBmWUm5Mk7PiWE63 R+CwFQ5DskGoYjkVuSdORwr4yTShB5lB49v7RBvvJBPRKhMvHFbXzu63yXIM2xLS 8mgRnFw1TAVquxzmYaGO/EbvPfe0ANi9kfx5sdgsKeR2s5UM= Received: from localhost (webms.mac.com [17.148.16.116]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a40.g.dreamhost.com (Postfix) with ESMTPSA id 8057F74C06E for ; Mon, 6 Dec 2010 12:34:44 -0800 (PST) To: user@cassandra.apache.org From: Aaron Morton Subject: Re: Re: Cassandra 0.7 beta 3 outOfMemory (OOM) Date: Mon, 06 Dec 2010 20:34:43 GMT X-Mailer: MobileMe Mail (1C321602) Message-id: <8bd6f189-1b57-6d24-fa9a-f1d7d1df7c25@me.com> Content-Type: multipart/alternative; boundary=Apple-Webmail-42--264b4eee-7457-c3f4-cd71-9e8284cb9814 MIME-Version: 1.0 In-Reply-To: <5c666dc0-7be2-6af6-dd8e-bf4a3b18cb4d@me.com> X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Webmail-42--264b4eee-7457-c3f4-cd71-9e8284cb9814 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Jake or anyone else got experience bulk loading into Lucandra ?=A0=0A=0AOr= does anyone have experience with JRocket ?=A0=0A=0AMax, are you sending o= ne document at a time into lucene. Can you send them in batches (like solr= ), if so does it reduce the=A0=0Aamount of requests going to cassandra?=A0= =0A=0AAlso, cassandra.bat is configured with=A0XX:+HeapDumpOnOutOfMemoryEr= ror so you should be able to take a look at where all the memory if going.= Riptano blog points to=A0http://www.eclipse.org/mat/=A0=A0also see=A0http= ://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gdyrr=0A=0A= Hope that helps.=A0=0A=0AAaron=0A=0AOn 07 Dec, 2010,at 09:17 AM, Aaron Mor= ton wrote:=0A=0AAccidentally=A0sent to me.=0A=0A= Begin forwarded message:=0AFrom: Max =0ADate: 07 Decem= ber 2010 6:00:36 AM=0ATo: Aaron Morton =0ASubject= : Re: Re: Re: Cassandra 0.7 beta 3 outOfMemory (OOM)=0A=0AThank you both f= or your answer!=0AAfter several tests with different parameters we came to= the =0Aconclusion that it must be a bug.=0AIt looks very similar to: http= s://issues.apache.org/jira/browse/CASSANDRA-1014=0A=0AFor both CFs we redu= ced thresholds:=0A- memtable_flush_after_mins =3D 60 (both CFs are used pe= rmanently, =0Atherefore other thresholds should trigger first)=0A- memtabl= e_throughput_in_mb =3D 40=0A- memtable_operations_in_millions =3D 0.3=0A- = keys_cached =3D 0=0A- rows_cached =3D 0=0A=0A- in_memory_compaction_limit_= in_mb =3D 64=0A=0AFirst we disabled caching, later we disabled compacting = and after that we set=0Acommitlog_sync: batch=0Acommitlog_sync_batch_windo= w_in_ms: 1=0A=0ABut our problem still appears:=0ADuring inserting files wi= th Lucandra memory usage is slowly growing =0Auntil OOM crash after about = 50 min.=0A@Peter: In our latest test we stopped writing suddenly but cassa= ndra =0Adidn\'t relax and remains even after minutes on ~90% heap usage.=0A= http://oi54.tinypic.com/2dueeix.jpg=0A=0AWith our heap calculation we shou= ld need:=0A64 MB * 2 * 3 + 1 GB =3D 1,4 GB=0AAll recent tests we run with = 3 GB. I think that should be ok for a =0Atest machine.=0AAlso consistency = level is one.=0A=0ABut Aaron is right, Lucandra produces even more than 20= 0 inserts/s.=0AMy 200 documents per second are about 200 operations (write= count) on =0Afirst CF and about 3000 on second CF.=0A=0ABut even with abou= t 120 documents/s cassandra crashes.=0A=0A=0ADisk I/O monitored with Windo= ws performance admin tools is on both =0Adiscs moderate (commitlog is on s= eperate harddisc).=0A=0A=0AAny ideas?=0AIf it's really a bug, in my opinio= n it's very critical.=0A=0A=0A=0AAaron Morton wr= ote:=0A=0A> I remember you have 2 CF's but what are the settings for:=A0=0A= >=0A> - memtable_flush_after_mins=0A> -=A0memtable_throughput_in_mb=0A> -=A0= memtable_operations_in_millions=0A> -=A0keys_cached=0A> -=A0rows_cached=0A= >=0A> -=A0in_memory_compaction_limit_in_mb=0A>=0A> Can you do the JVM Heap= Calculation here and see what it says=0A> http://wiki.apache.org/cassandr= a/MemtableThresholds=0A>=0A> What Consistency Level are you writing at? (C= hecking =A0it's not Zero)=A0=0A>=0A> When you talk about 200 inserts per s= econd is that storing 200 =0A> documents through lucandra or 200 request t= o cassandra. If it's the =0A> first option I would assume that would gener= ate a lot more actual =0A> requests into cassandra. Open up jconsole and t= ake a look at the =0A> WriteCount settings for the =0A> CF's=A0http://wiki= apache.org/cassandra/MemtableThresholds=0A>=0A> You could also try settin= g the compaction thresholds to 0 to disable=0A> compaction while you are p= ushing this data in. Then use node tool to=0A> compact and turn the settin= gs back to normal. See cassandra.yam for=0A> more info.=0A>=0A> I would ha= ve thought you could get the writes through with the setup=0A> you've desc= ribed so far (even though a single 32bit node is unusual).=0A> The best ad= vice is to turn all the settings down (e.g. caches off,=0A> mtable flush 6= 4MB, compaction disabled) and if it still fails try:=0A>=0A> - checking yo= ur IO stats, not sure on windows but JConsole has some IO=0A> stats. If yo= ur IO cannot keep up then your server is not fast enough=0A> for your clie= nt load.=0A> - reducing the client load=0A>=0A> Hope that helps.=A0=0A> Aa= ron=0A>=0A>=0A> On 04 Dec, 2010,at 05:23 AM, Max wrot= e:=0A>=0A> Hi,=0A>=0A> we increased heap space to 3 GB (with JRocket VM un= der 32-bit Win with=0A> 4 GB RAM)=0A> but under "heavy" inserts Cassandra = is still crashing with OutOfMemory=0A> error after a GC storm.=0A>=0A> It = sounds very similar to =0A> https://issues.apache.org/jira/browse/CASSANDR= A-1177=0A>=0A> In our insert-tests the average heap usage is slowly growin= g up to the=0A> 3 GB border (jconsole monitor over 50 min=0A> http://oi51.= tinypic.com/k12gzd.jpg) and the CompactionManger queue is=0A> also constan= tly growing up to about 50 jobs pending=0A>=0A> We tried to decrease CF me= mtable threshold but after about half a=0A> million inserts it's over.=0A>= =0A> - Cassandra 0.7.0 beta 3=0A> - Single Node=0A> - about 200 inserts/s = ~500byte - 1 kb=0A>=0A>=0A> Is there no other possibility instead of slowi= ng down inserts/s ?=0A>=0A> What could be an indicator to see if a node wo= rks stable with this=0A> amount of inserts?=0A>=0A> Thank you for your ans= wer,=0A> Max=0A>=0A>=0A> Aaron Morton :=0A>=0A>> = Sounds like you need to increase the Heap size and/or reduce the =0A>> mem= table_throughput_in_mb and/or turn off the internal caches. =0A>> Normally= the binary memtable thresholds only apply to bulk load =0A>> operations a= nd it's the per CF memtable_* settings you want to =0A>> change. I'm not=A0= familiar=A0with lucandra though.=A0=0A>>=0A>> See the section on JVM Heap = Size here=A0=0A>> http://wiki.apache.org/cassandra/MemtableThresholds=0A>>= =0A>> Bottom line is you will need more JVM heap memory.=0A>>=0A>> Hope th= at helps.=0A>> Aaron=0A>>=0A>> On 29 Nov, 2010,at 10:28 PM, cassandra@ajow= a.de wrote:=0A>>=0A>> Hi community,=0A>>=0A>> during my tests i had severa= l OOM crashes.=0A>> Getting some hints to find out the problem would be ni= ce.=0A>>=0A>> First cassandra crashes after about 45 min insert test scrip= t.=0A>> During the following tests time to OOM was shorter until it starte= d to crash=0A>> even in "idle" mode.=0A>>=0A>> Here the facts:=0A>> - cass= andra 0.7 beta 3=0A>> - using lucandra to index about 3 million files ~1kb= data=0A>> - inserting with one client to one cassandra node with about 20= 0 files/s=0A>> - cassandra data files for this keyspace grow up to about 2= 0 GB=0A>> - the keyspace only contains the two lucandra specific CFs=0A>>=0A= >> Cluster:=0A>> - cassandra single node on windows 32bit, Xeon 2,5 Ghz, 4= GB RAM=0A>> - java jre 1.6.0_22=0A>> - heap space first 1GB, later increas= ed to 1,3 GB=0A>>=0A>> Cassandra.yaml:=0A>> default + reduced "binary_memt= able_throughput_in_mb" to 128=0A>>=0A>> CFs:=0A>> default + reduced=0A>> m= in_compaction_threshold: 4=0A>> max_compaction_threshold: 8=0A>>=0A>>=0A>>= I think the problem appears always during compaction,=0A>> and perhaps it= is a result of large rows (some about 170mb).=0A>>=0A>> Are there more op= tions we could use to work with few memory?=0A>>=0A>> Is it a problem of c= ompaction?=0A>> And how to avoid?=0A>> Slower inserts? More memory?=0A>> E= ven fewer memtable_throuput or in_memory_compaction_limit?=0A>> Continuous= manual major comapction?=0A>>=0A>> I've read=0A>> http://www.riptano.com/= docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors=0A>> - row_= size should be fixed since 0.7 and 200mb is still far away from 2gb=0A>> -= only key cache is used a little bit 3600/20000=0A>> - after a lot of writ= es cassandra crashes even in idle mode=0A>> - memtablesize was reduced and= there are only 2 CFs=0A>>=0A>> Several heapdumps in MAT show 60-99% heapu= sage of compaction thread.=0A --Apple-Webmail-42--264b4eee-7457-c3f4-cd71-9e8284cb9814 Content-Type: multipart/related; type="text/html"; boundary=Apple-Webmail-86--264b4eee-7457-c3f4-cd71-9e8284cb9814 --Apple-Webmail-86--264b4eee-7457-c3f4-cd71-9e8284cb9814 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1;
Jake or anyone else got experience bulk loading = into Lucandra ? 

Or does anyone have experie= nce with JRocket ? 

Max, are you sending one= document at a time into lucene. Can you send them in batches (like solr),= if so does it reduce the 
amount of requests going to cass= andra? 

Also, cassandra.bat is configured wi= th XX:+HeapDumpOnOutOfMemoryError so you should be able to take a loo= k at where all the memory if going. Riptano blog points to http://www.eclipse.org/mat/  a= lso see http://www.oracle.com/technetwork/java/javase/meml= eaks-137499.html#gdyrr

Ho= pe that helps. 

Aaron

On 07 De= c, 2010,at 09:17 AM, Aaron Morton <aaron@thelastpickle.com> wrote:
Accidentally sen= t to me.

Begin forwarded message:
From: Max <cassandra@ajowa.de>
Date: 07 December 2010 6:00:36 AM
To: Aaron Morton= <aaron@thelastpickle.com>
Subject: Re: Re: Re: Cassa= ndra 0.7 beta 3 outOfMemory (OOM)

Thank you both for your answer!
=0AAfter several tests with different= parameters we came to the
=0Aconclusion that it must be a bug.
=0A= It looks very similar to: https://issues.apache.org/jira/browse/CASSANDRA-1014
=0A<= br>=0AFor both CFs we reduced thresholds:
=0A- memtable_flush_after_min= s =3D 60 (both CFs are used permanently,
=0Atherefore other threshold= s should trigger first)
=0A- memtable_throughput_in_mb =3D 40
=0A- m= emtable_operations_in_millions =3D 0.3
=0A- keys_cached =3D 0
=0A- r= ows_cached =3D 0
=0A
=0A- in_memory_compaction_limit_in_mb =3D 64=0A
=0AFirst we disabled caching, later we disabled compacting and aft= er that we set
=0Acommitlog_sync: batch
=0Acommitlog_sync_batch_wind= ow_in_ms: 1
=0A
=0ABut our problem still appears:
=0ADuring inser= ting files with Lucandra memory usage is slowly growing
=0Auntil OOM = crash after about 50 min.
=0A@Peter: In our latest test we stopped writ= ing suddenly but cassandra
=0Adidn\'t relax and remains even after mi= nutes on ~90% heap usage.
=0Ahttp://oi54.tinypi= c.com/2dueeix.jpg
=0A
=0AWith our heap calculation we should nee= d:
=0A64 MB * 2 * 3 + 1 GB =3D 1,4 GB
=0AAll recent tests we run wit= h 3 GB. I think that should be ok for a
=0Atest machine.
=0AAlso c= onsistency level is one.
=0A
=0ABut Aaron is right, Lucandra produce= s even more than 200 inserts/s.
=0AMy 200 documents per second are abou= t 200 operations (writecount) on
=0Afirst CF and about 3000 on second= CF.
=0A
=0ABut even with about 120 documents/s cassandra crashes.=0A
=0A
=0ADisk I/O monitored with Windows performance admin tools= is on both
=0Adiscs moderate (commitlog is on seperate harddisc).=0A
=0A
=0AAny ideas?
=0AIf it's really a bug, in my opinion it'= s very critical.
=0A
=0A
=0A
=0AAaron Morton <aaron@thelast= pickle.com> wrote:
=0A
=0A> I remember you have 2 CF's but wha= t are the settings for: 
=0A>
=0A> - memtable_flush_after= _mins
=0A> - memtable_throughput_in_mb
=0A> - memtab= le_operations_in_millions
=0A> - keys_cached
=0A> - = rows_cached
=0A>
=0A> - in_memory_compaction_limit_in_mb<= br>=0A>
=0A> Can you do the JVM Heap Calculation here and see wha= t it says
=0A> http://wiki.apache.org/cassandra/MemtableThresholds
=0A>
=0A= > What Consistency Level are you writing at? (Checking  it's not Z= ero) 
=0A>
=0A> When you talk about 200 inserts per secon= d is that storing 200
=0A> documents through lucandra or 200 requ= est to cassandra. If it's the
=0A> first option I would assume th= at would generate a lot more actual
=0A> requests into cassandra.= Open up jconsole and take a look at the
=0A> WriteCount settings= for the
=0A> CF's http://wiki.apache.org/cassandra/MemtableThresholds
= =0A>
=0A> You could also try setting the compaction thresholds to= 0 to disable
=0A> compaction while you are pushing this data in. Th= en use node tool to
=0A> compact and turn the settings back to norma= l. See cassandra.yam for
=0A> more info.
=0A>
=0A> I wou= ld have thought you could get the writes through with the setup
=0A>= you've described so far (even though a single 32bit node is unusual).
= =0A> The best advice is to turn all the settings down (e.g. caches off,=
=0A> mtable flush 64MB, compaction disabled) and if it still fails = try:
=0A>
=0A> - checking your IO stats, not sure on windows b= ut JConsole has some IO
=0A> stats. If your IO cannot keep up then y= our server is not fast enough
=0A> for your client load.
=0A> = - reducing the client load
=0A>
=0A> Hope that helps. =0A> Aaron
=0A>
=0A>
=0A> On 04 Dec, 2010,at 05:23 A= M, Max <cassandra@ajowa.de> wrote:
=0A>
=0A> Hi,
=0A&= gt;
=0A> we increased heap space to 3 GB (with JRocket VM under 32-b= it Win with
=0A> 4 GB RAM)
=0A> but under "heavy" inserts Cass= andra is still crashing with OutOfMemory
=0A> error after a GC storm=
=0A>
=0A> It sounds very similar to
=0A> https://issues.apache.org/= jira/browse/CASSANDRA-1177
=0A>
=0A> In our insert-tests t= he average heap usage is slowly growing up to the
=0A> 3 GB border (= jconsole monitor over 50 min
=0A> http://oi51.= tinypic.com/k12gzd.jpg) and the CompactionManger queue is
=0A> a= lso constantly growing up to about 50 jobs pending
=0A>
=0A> W= e tried to decrease CF memtable threshold but after about half a
=0A>= ; million inserts it's over.
=0A>
=0A> - Cassandra 0.7.0 beta = 3
=0A> - Single Node
=0A> - about 200 inserts/s ~500byte - 1 k= b
=0A>
=0A>
=0A> Is there no other possibility instead o= f slowing down inserts/s ?
=0A>
=0A> What could be an indicato= r to see if a node works stable with this
=0A> amount of inserts?=0A>
=0A> Thank you for your answer,
=0A> Max
=0A>=0A>
=0A> Aaron Morton <aaron@thelastpickle.com>:
=0A&= gt;
=0A>> Sounds like you need to increase the Heap size and/or r= educe the
=0A>> memtable_throughput_in_mb and/or turn off the = internal caches.
=0A>> Normally the binary memtable thresholds= only apply to bulk load
=0A>> operations and it's the per CF = memtable_* settings you want to
=0A>> change. I'm not fam= iliar with lucandra though. 
=0A>>
=0A>> See t= he section on JVM Heap Size here 
=0A>> http://wiki.apache.org/cassandra/Memta= bleThresholds
=0A>>
=0A>> Bottom line is you will ne= ed more JVM heap memory.
=0A>>
=0A>> Hope that helps.=0A>> Aaron
=0A>>
=0A>> On 29 Nov, 2010,at 10:28 = PM, cassandra@ajowa.de wrote:
=0A>>
=0A>> Hi community,<= br>=0A>>
=0A>> during my tests i had several OOM crashes.=0A>> Getting some hints to find out the problem would be nice.=0A>>
=0A>> First cassandra crashes after about 45 min ins= ert test script.
=0A>> During the following tests time to OOM was= shorter until it started to crash
=0A>> even in "idle" mode.
= =0A>>
=0A>> Here the facts:
=0A>> - cassandra 0.7 = beta 3
=0A>> - using lucandra to index about 3 million files ~1kb= data
=0A>> - inserting with one client to one cassandra node wit= h about 200 files/s
=0A>> - cassandra data files for this keyspac= e grow up to about 20 GB
=0A>> - the keyspace only contains the t= wo lucandra specific CFs
=0A>>
=0A>> Cluster:
=0A>= > - cassandra single node on windows 32bit, Xeon 2,5 Ghz, 4GB RAM
=0A= >> - java jre 1.6.0_22
=0A>> - heap space first 1GB, later = increased to 1,3 GB
=0A>>
=0A>> Cassandra.yaml:
=0A&g= t;> default + reduced "binary_memtable_throughput_in_mb" to 128
=0A&= gt;>
=0A>> CFs:
=0A>> default + reduced
=0A>>= ; min_compaction_threshold: 4
=0A>> max_compaction_threshold: 8=0A>>
=0A>>
=0A>> I think the problem appears al= ways during compaction,
=0A>> and perhaps it is a result of large= rows (some about 170mb).
=0A>>
=0A>> Are there more opt= ions we could use to work with few memory?
=0A>>
=0A>> I= s it a problem of compaction?
=0A>> And how to avoid?
=0A>&= gt; Slower inserts? More memory?
=0A>> Even fewer memtable_throup= ut or in_memory_compaction_limit?
=0A>> Continuous manual major c= omapction?
=0A>>
=0A>> I've read
=0A>> http://www.riptano.com/docs/0.6/tr= oubleshooting/index#nodes-are-dying-with-oom-errors
=0A>> - r= ow_size should be fixed since 0.7 and 200mb is still far away from 2gb
= =0A>> - only key cache is used a little bit 3600/20000
=0A>>= ; - after a lot of writes cassandra crashes even in idle mode
=0A>&g= t; - memtablesize was reduced and there are only 2 CFs
=0A>>
=0A= >> Several heapdumps in MAT show 60-99% heapusage of compaction thre= ad.
=0A
--Apple-Webmail-86--264b4eee-7457-c3f4-cd71-9e8284cb9814-- --Apple-Webmail-42--264b4eee-7457-c3f4-cd71-9e8284cb9814--