From user-return-10808-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Nov 29 21:03:27 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 40015 invoked from network); 29 Nov 2010 21:03:27 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 29 Nov 2010 21:03:27 -0000 Received: (qmail 65105 invoked by uid 500); 29 Nov 2010 21:03:24 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 65081 invoked by uid 500); 29 Nov 2010 21:03:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 65073 invoked by uid 99); 29 Nov 2010 21:03:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 21:03:24 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a54.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 21:03:16 +0000 Received: from homiemail-a54.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTP id 5A03B3A406B for ; Mon, 29 Nov 2010 13:02:54 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=to:from :subject:date:message-id:content-type:mime-version:in-reply-to; q=dns; s=thelastpickle.com; b=afyBQ6yJukE3OqABphr/qA9MAlnABYh1v kZVTAPm+vwc3yoTOlCIwdS3bYRBt8I+Yq1wodpP9M6Gsb4rgpDlwfkE9KFmJPca2 wepvrTpT3cfKvq3d7aOM7D4bh4PhIiI42hidjtOkf0z+maWrD5UtmW4Qb8F9tv9o TByiPmPXWc= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=to :from:subject:date:message-id:content-type:mime-version: in-reply-to; s=thelastpickle.com; bh=ZRK4yxPCGDQ1Hddb4ZxktKbKNmA =; b=A10B2l1xbkyxjybF+gIsznrn/AJgRZcEBpqnofk2TJkE+Ju7LaujhtjNUkq c596prVfppdnIkeFWZPO6Lybkh4UyqhjQHt7LcpqdLzTNC2qSG236llFPWlO/7fx P/Ru948HG5RRjSGqax8bWgKDciwBOuUWDMo1uDNfwwUNQ2J0= Received: from localhost (webms.mac.com [17.148.16.116]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTPSA id 33B2B3A405B for ; Mon, 29 Nov 2010 13:02:54 -0800 (PST) To: user@cassandra.apache.org From: Aaron Morton Subject: Re: Cassandra 0.7 beta 3 outOfMemory (OOM) Date: Mon, 29 Nov 2010 21:02:53 GMT X-Mailer: MobileMe Mail (1C3207) Message-id: <107a4ba3-d8f9-4be0-cb5f-9d999d3684b3@me.com> Content-Type: multipart/alternative; boundary=Apple-Webmail-42--71250b74-6267-6315-9700-ffab52b71c77 MIME-Version: 1.0 In-Reply-To: <20101129102829.m7vd00jg744so4wc@webmail.planet-school.de> X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Webmail-42--71250b74-6267-6315-9700-ffab52b71c77 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Sounds like you need to increase the Heap size and/or reduce the memtable_= throughput_in_mb and/or turn off the internal caches. Normally the binary = memtable thresholds only apply to bulk load operations and it's the per CF= memtable_* settings you want to change. I'm not=A0familiar=A0with lucandr= a though.=A0=0A=0ASee the section on JVM Heap Size here=A0=0Ahttp://wiki.a= pache.org/cassandra/MemtableThresholds=0A=0ABottom line is you will need m= ore JVM heap memory.=0A=0AHope that helps.=0AAaron=0A=0AOn 29 Nov, 2010,at= 10:28 PM, cassandra@ajowa.de wrote:=0A=0AHi community,=0A=0Aduring my tes= ts i had several OOM crashes.=0AGetting some hints to find the problem wou= ld be nice.=0A=0AFirst cassandra crashes after about 45 min insert test sc= ript.=0ADuring the following tests time to OOM was shorter until it starte= d to crash=0Aeven in "idle" mode.=0A=0AHere the facts:=0A- cassandra 0.7 b= eta 3=0A- using lucandra to index about 3 million files ~1kb data=0A- inse= rting with one client to one cassandra node with about 200 files/s=0A- cas= sandra data files for this keyspace grow up to about 20 GB=0A- the keyspac= e only contains the two lucandra specific CFs=0A=0ACluster:=0A- cassandra = single node on windows 32bit, Xeon 2,5 Ghz, 4GB RAM=0A- java jre 1.6.0_22=0A= - heap space first 1GB, later increased to 1,3 GB=0A=0ACassandra.yaml:=0Ad= efault + reduced "binary_memtable_throughput_in_mb" to 128=0A=0ACFs:=0Adef= ault + reduced=0Amin_compaction_threshold: 4=0Amax_compaction_threshold: 8= =0A=0A=0AI think the problem appears always during compaction,=0Aand perha= ps it is a result of large rows (some about 170mb).=0A=0AAre there more op= tionions we could use to work with few memory?=0A=0AIs it a problem of com= paction?=0AAnd how to avoid?=0ASlower inserts? More memory?=0AEven fewer m= emtable_throuput or in_memory_compaction_limit?=0AContinuous manual major = comapction?=0A=0AI've read =0Ahttp://www.riptano.com/docs/0.6/troubleshoot= ing/index#nodes-are-dying-with-oom-errors=0A- row_size should be fixed sin= ce 0.7 and 200mb is still far away from 2gb=0A- only key cache is used a l= ittle bit 3600/20000=0A- after a lot of writes cassandra crashes even in i= dle mode=0A- memtablesize was reduced and there are only 2 CFs=0A=0ASevera= l heapdumps in MAT show 60-99% heapusage of compaction thread.=0A=0AHere s= ome log extract:=0A=0AINFO [CompactionExecutor:1] 2010-11-26 14:18:18,593 = =0ACompactionIterator.java (line 134) Compacting large row =0A665032557271= 7566efbfbf44545241434b53efbfbf31 (172967291 bytes) =0Aincrementally=0AINFO= [ScheduledTasks:1] 2010-11-26 14:18:41,421 GCInspector.java =0A(line 133)= GC for ParNew: 365 ms, 54551328 reclaimed leaving 459496840 =0Aused; max = is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:42,437 GCInspecto= r.java =0A(line 133) GC for ParNew: 226 ms, 12469104 reclaimed leaving 554= 506776 =0Aused; max is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-26 14:= 18:43,453 GCInspector.java =0A(line 133) GC for ParNew: 224 ms, 12777840 r= eclaimed leaving 649207976 =0Aused; max is 1450442752=0AINFO [ScheduledTas= ks:1] 2010-11-26 14:18:44,468 GCInspector.java =0A(line 133) GC for ParNew= : 225 ms, 12564144 reclaimed leaving 744122872 =0Aused; max is 1450442752=0A= INFO [ScheduledTasks:1] 2010-11-26 14:18:45,468 GCInspector.java =0A(line = 133) GC for ParNew: 222 ms, 16020328 reclaimed leaving 835581584 =0Aused; = max is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:46,468 GCInsp= ector.java =0A(line 133) GC for ParNew: 226 ms, 12697912 reclaimed leaving= 930362712 =0Aused; max is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-26= 14:18:47,468 GCInspector.java =0A(line 133) GC for ParNew: 227 ms, 158168= 72 reclaimed leaving =0A1022026288 used; max is 1450442752=0AINFO [Schedul= edTasks:1] 2010-11-26 14:18:48,484 GCInspector.java =0A(line 133) GC for P= arNew: 258 ms, 12746584 reclaimed leaving =0A1116758744 used; max is 14504= 42752=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:49,484 GCInspector.java =0A= (line 133) GC for ParNew: 257 ms, 12802608 reclaimed leaving =0A1211435176= used; max is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,546= GCInspector.java =0A(line 133) GC for ConcurrentMarkSweep: 4188 ms, 27130= 8512 reclaimed =0Aleaving 1047605704 used; max is 1450442752=0AINFO [Sched= uledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java =0A(line 153) Pool N= ame Active Pending=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,546 GCIns= pector.java =0A(line 160) ResponseStage 0 0=0AINFO [ScheduledTasks:1] 2010= -11-26 14:18:54,546 GCInspector.java =0A(line 160) ReadStage 0 0=0AINFO [S= cheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java =0A(line 160) Re= adRepair 0 0=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector= java =0A(line 160) MutationStage 0 0=0AINFO [ScheduledTasks:1] 2010-11-26= 14:18:54,546 GCInspector.java =0A(line 160) GossipStage 0 0=0AINFO [Sched= uledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java =0A(line 160) Antien= tropyStage 0 0=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspect= or.java =0A(line 160) MigrationStage 0 0=0AINFO [ScheduledTasks:1] 2010-11= -26 14:18:54,562 GCInspector.java =0A(line 160) StreamStage 0 0=0AINFO [Sc= heduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java =0A(line 160) Mem= tablePostFlusher 0 0=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCI= nspector.java =0A(line 160) FlushWriter 0 0=0AINFO [ScheduledTasks:1] 2010= -11-26 14:18:54,562 GCInspector.java =0A(line 160) MiscStage 0 0=0AINFO [S= cheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java =0A(line 160) Fl= ushSorter 0 0=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspecto= r.java =0A(line 164) CompactionManager n/a 6=0AINFO [ScheduledTasks:1] 201= 0-11-26 14:18:54,562 GCInspector.java =0A(line 176) MessagingService n/a 0= ,0=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java =0A(= line 180) ColumnFamily Memtable ops,data Row cache =0Asize/cap Key cache s= ize/cap=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java= =0A(line 183) system.LocationInfo 1,17 =0A0/0 3/3=0AINFO [ScheduledTasks:= 1] 2010-11-26 14:18:54,562 GCInspector.java =0A(line 183) system.HintsColu= mnFamily 0,0 =0A0/0 0/1=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,578 = GCInspector.java =0A(line 183) system.Migrations 0,0 =0A0/0 0/2=0AINFO [Sc= heduledTasks:1] 2010-11-26 14:18:54,578 GCInspector.java =0A(line 183) sys= tem.Schema 0,0 =0A0/0 3/3=0AINFO [ScheduledTasks:1] 2010-11-26 14:18:54,57= 8 GCInspector.java =0A(line 183) system.IndexInfo 0,0 =0A0/0 0/1=0AINFO [S= cheduledTasks:1] 2010-11-26 14:18:54,578 GCInspector.java =0A(line 183) Lu= candra.Documents 0,0 =0A0/0 0/200000=0AINFO [ScheduledTasks:1] 2010-11-26 = 14:18:54,578 GCInspector.java =0A(line 183) Lucandra.TermInfo 0,0 =0A0/0 0= /200000=0AINFO [ScheduledTasks:1] 2010-11-26 14:19:05,328 GCInspector.java= =0A(line 133) GC for ConcurrentMarkSweep: 3807 ms, 31040 reclaimed =0Alea= ving 1153889144 used; max is 1450442752=0AINFO [ScheduledTasks:1] 2010-11-= 26 14:20:09,093 GCInspector.java =0A(line 153) Pool Name Active Pending=0A= INFO [ScheduledTasks:1] 2010-11-26 14:20:09,093 GCInspector.java =0A(line = 160) ResponseStage 0 0=0AERROR [CompactionExecutor:1] 2010-11-26 14:20:09,= 093 =0AAbstractCassandraDaemon.java (line 89) Fatal exception in thread =0A= Thread[CompactionExecutor:1,1,main]=0Ajava.lang.OutOfMemoryError: Java hea= p space=0Aat java.util.Arrays.copyOf(Arrays.java:2786)=0Aat java.io.ByteAr= rayOutputStream.write(ByteArrayOutputStream.java:94)=0Aat java.io.DataOutp= utStream.writeLong(DataOutputStream.java:207)=0Aat =0Aorg.apache.cassandra= db.SuperColumnSerializer.serialize(SuperColumn.java:312)=0Aat =0Aorg.apac= he.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:291)=0Aat= =0Aorg.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(Col= umnFamilySerializer.java:82)=0Aat =0Aorg.apache.cassandra.db.ColumnFamilyS= erializer.serializeWithIndexes(ColumnFamilySerializer.java:101)=0Aat org.a= pache.cassandra.io.PrecompactedRow.(PrecompactedRow.java:96)=0Aat =0A= org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionItera= tor.java:138)=0Aat =0Aorg.apache.cassandra.io.CompactionIterator.getReduce= d(CompactionIterator.java:107)=0Aat =0Aorg.apache.cassandra.io.CompactionI= terator.getReduced(CompactionIterator.java:42)=0Aat =0Aorg.apache.cassandr= a.utils.ReducingIterator.computeNext(ReducingIterator.java:73)=0Aat =0Acom= google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.= java:136)=0Aat =0Acom.google.common.collect.AbstractIterator.hasNext(Abstr= actIterator.java:131)=0Aat =0Aorg.apache.commons.collections.iterators.Fil= terIterator.setNextObject(FilterIterator.java:183)=0Aat =0Aorg.apache.comm= ons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)=0A= at =0Aorg.apache.cassandra.db.CompactionManager.doCompaction(CompactionMan= ager.java:273)=0Aat =0Aorg.apache.cassandra.db.CompactionManager$1.call(Co= mpactionManager.java:109)=0Aat =0Aorg.apache.cassandra.db.CompactionManage= r$1.call(CompactionManager.java:87)=0Aat java.util.concurrent.FutureTask$S= ync.innerRun(FutureTask.java:303)=0Aat java.util.concurrent.FutureTask.run= (FutureTask.java:138)=0Aat =0Ajava.util.concurrent.ThreadPoolExecutor$Work= er.runTask(ThreadPoolExecutor.java:886)=0Aat =0Ajava.util.concurrent.Threa= dPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)=0Aat java.lang.Threa= d.run(Thread.java:662)=0A --Apple-Webmail-42--71250b74-6267-6315-9700-ffab52b71c77 Content-Type: multipart/related; type="text/html"; boundary=Apple-Webmail-86--71250b74-6267-6315-9700-ffab52b71c77 --Apple-Webmail-86--71250b74-6267-6315-9700-ffab52b71c77 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=ISO-8859-1;
Sounds like you need to increase the Heap size and/or reduce the memt= able_throughput_in_mb and/or turn off the internal caches. Normally the bi= nary memtable thresholds only apply to bulk load operations and it's the p= er CF memtable_* settings you want to change. I'm not familiar w= ith lucandra though. 

See the section on JVM= Heap Size here 

Bott= om line is you will need more JVM heap memory.

Ho= pe that helps.
Aaron

On 29 Nov, 2010,at 10:28 PM, cassand= ra@ajowa.de wrote:

Hi community,
=0A
=0Aduring my tests i had several= OOM crashes.
=0AGetting some hints to find the problem would be nice.<= br>=0A
=0AFirst cassandra crashes after about 45 min insert test script=
=0ADuring the following tests time to OOM was shorter until it starte= d to crash
=0Aeven in "idle" mode.
=0A
=0AHere the facts:
=0A-= cassandra 0.7 beta 3
=0A- using lucandra to index about 3 million file= s ~1kb data
=0A- inserting with one client to one cassandra node with a= bout 200 files/s
=0A- cassandra data files for this keyspace grow up to= about 20 GB
=0A- the keyspace only contains the two lucandra specific = CFs
=0A
=0ACluster:
=0A- cassandra single node on windows 32bit, = Xeon 2,5 Ghz, 4GB RAM
=0A- java jre 1.6.0_22
=0A- heap space first 1= GB, later increased to 1,3 GB
=0A
=0ACassandra.yaml:
=0Adefault += reduced "binary_memtable_throughput_in_mb" to 128
=0A
=0ACFs:
=0A= default + reduced
=0Amin_compaction_threshold: 4
=0Amax_compaction_t= hreshold: 8
=0A
=0A
=0AI think the problem appears always during = compaction,
=0Aand perhaps it is a result of large rows (some about 170= mb).
=0A
=0AAre there more optionions we could use to work with few = memory?
=0A
=0AIs it a problem of compaction?
=0AAnd how to avoid= ?
=0ASlower inserts? More memory?
=0AEven fewer memtable_throuput or= in_memory_compaction_limit?
=0AContinuous manual major comapction?
= =0A
=0AI've read
=0Ahttp://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-= with-oom-errors
=0A- row_size should be fixed since 0.7 and 200mb i= s still far away from 2gb
=0A- only key cache is used a little bit 3600= /20000
=0A- after a lot of writes cassandra crashes even in idle mode=0A- memtablesize was reduced and there are only 2 CFs
=0A
=0ASeve= ral heapdumps in MAT show 60-99% heapusage of compaction thread.
=0A=0AHere some log extract:
=0A
=0A INFO [CompactionExecutor:1] 2010= -11-26 14:18:18,593
=0ACompactionIterator.java (line 134) Compacting = large row
=0A6650325572717566efbfbf44545241434b53efbfbf31 (172967291 = bytes)
=0Aincrementally
=0A INFO [ScheduledTasks:1] 2010-11-26 14= :18:41,421 GCInspector.java
=0A(line 133) GC for ParNew: 365 ms, 5455= 1328 reclaimed leaving 459496840
=0Aused; max is 1450442752
=0A I= NFO [ScheduledTasks:1] 2010-11-26 14:18:42,437 GCInspector.java
=0A(l= ine 133) GC for ParNew: 226 ms, 12469104 reclaimed leaving 554506776
= =0Aused; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-11-26 14:1= 8:43,453 GCInspector.java
=0A(line 133) GC for ParNew: 224 ms, 127778= 40 reclaimed leaving 649207976
=0Aused; max is 1450442752
=0A INF= O [ScheduledTasks:1] 2010-11-26 14:18:44,468 GCInspector.java
=0A(lin= e 133) GC for ParNew: 225 ms, 12564144 reclaimed leaving 744122872
=0A= used; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:4= 5,468 GCInspector.java
=0A(line 133) GC for ParNew: 222 ms, 16020328 = reclaimed leaving 835581584
=0Aused; max is 1450442752
=0A INFO [= ScheduledTasks:1] 2010-11-26 14:18:46,468 GCInspector.java
=0A(line 1= 33) GC for ParNew: 226 ms, 12697912 reclaimed leaving 930362712
=0Aus= ed; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:47,= 468 GCInspector.java
=0A(line 133) GC for ParNew: 227 ms, 15816872 re= claimed leaving
=0A1022026288 used; max is 1450442752
=0A INFO [S= cheduledTasks:1] 2010-11-26 14:18:48,484 GCInspector.java
=0A(line 13= 3) GC for ParNew: 258 ms, 12746584 reclaimed leaving
=0A1116758744 us= ed; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:49,= 484 GCInspector.java
=0A(line 133) GC for ParNew: 257 ms, 12802608 re= claimed leaving
=0A1211435176 used; max is 1450442752
=0A INFO [S= cheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java
=0A(line 13= 3) GC for ConcurrentMarkSweep: 4188 ms, 271308512 reclaimed
=0Aleavin= g 1047605704 used; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-= 11-26 14:18:54,546 GCInspector.java
=0A(line 153) Pool Name = Active Pending
=0A INFO [ScheduledTasks:1] 2010-11-26 14:1= 8:54,546 GCInspector.java
=0A(line 160) ResponseStage = 0 0
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,546 G= CInspector.java
=0A(line 160) ReadStage 0 = 0
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector= java
=0A(line 160) ReadRepair 0 0
= =0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java =0A(line 160) MutationStage 0 0
=0A INFO = [ScheduledTasks:1] 2010-11-26 14:18:54,546 GCInspector.java
=0A(line = 160) GossipStage 0 0
=0A INFO [Scheduled= Tasks:1] 2010-11-26 14:18:54,546 GCInspector.java
=0A(line 160) Antie= ntropyStage 0 0
=0A INFO [ScheduledTasks:1] 2= 010-11-26 14:18:54,562 GCInspector.java
=0A(line 160) MigrationStage = 0 0
=0A INFO [ScheduledTasks:1] 2010-11-26 = 14:18:54,562 GCInspector.java
=0A(line 160) StreamStage = 0 0
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,5= 62 GCInspector.java
=0A(line 160) MemtablePostFlusher 0= 0
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspe= ctor.java
=0A(line 160) FlushWriter 0 0=
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java =
=0A(line 160) MiscStage 0 0
=0A I= NFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java
=0A(l= ine 160) FlushSorter 0 0
=0A INFO [Sched= uledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java
=0A(line 164) C= ompactionManager n/a 6
=0A INFO [ScheduledTasks:= 1] 2010-11-26 14:18:54,562 GCInspector.java
=0A(line 176) MessagingSe= rvice n/a 0,0
=0A INFO [ScheduledTasks:1] 2010-11= -26 14:18:54,562 GCInspector.java
=0A(line 180) ColumnFamily = Memtable ops,data Row cache
=0Asize/cap Key cache size/cap=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,562 GCInspector.java <= br>=0A(line 183) system.LocationInfo 1,17 =
=0A 0/0 3/3
=0A INFO [ScheduledTasks:1] 2010= -11-26 14:18:54,562 GCInspector.java
=0A(line 183) system.HintsColumn= Family 0,0
=0A 0/0 0= /1
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,578 GCInspector.jav= a
=0A(line 183) system.Migrations 0,0 =
=0A 0/0 0/2
=0A INFO [ScheduledTasks:1] = 2010-11-26 14:18:54,578 GCInspector.java
=0A(line 183) system.Schema = 0,0
=0A 0/0 = 3/3
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,578 GCInspector= java
=0A(line 183) system.IndexInfo 0,0 =
=0A 0/0 0/1
=0A INFO [ScheduledTasks= :1] 2010-11-26 14:18:54,578 GCInspector.java
=0A(line 183) Lucandra.D= ocuments 0,0
=0A 0/0 = 0/200000
=0A INFO [ScheduledTasks:1] 2010-11-26 14:18:54,578 GCInspe= ctor.java
=0A(line 183) Lucandra.TermInfo 0,0=
=0A 0/0 0/200000
=0A INFO [ScheduledT= asks:1] 2010-11-26 14:19:05,328 GCInspector.java
=0A(line 133) GC for= ConcurrentMarkSweep: 3807 ms, 31040 reclaimed
=0Aleaving 1153889144 = used; max is 1450442752
=0A INFO [ScheduledTasks:1] 2010-11-26 14:20:0= 9,093 GCInspector.java
=0A(line 153) Pool Name Act= ive Pending
=0A INFO [ScheduledTasks:1] 2010-11-26 14:20:09,093 GCIn= spector.java
=0A(line 160) ResponseStage 0 = 0
=0AERROR [CompactionExecutor:1] 2010-11-26 14:20:09,093
=0AAbs= tractCassandraDaemon.java (line 89) Fatal exception in thread
=0AThre= ad[CompactionExecutor:1,1,main]
=0Ajava.lang.OutOfMemoryError: Java hea= p space
=0A at java.util.Arrays.copyOf(Arrays.java:2786)
=0A at java= io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
=0A at j= ava.io.DataOutputStream.writeLong(DataOutputStream.java:207)
=0A at =0Aorg.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.j= ava:312)
=0A at
=0Aorg.apache.cassandra.db.SuperColumnSerializer.s= erialize(SuperColumn.java:291)
=0A at
=0Aorg.apache.cassandra.db.C= olumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:82)<= br>=0A at
=0Aorg.apache.cassandra.db.ColumnFamilySerializer.serialize= WithIndexes(ColumnFamilySerializer.java:101)
=0A at org.apache.cassandr= a.io.PrecompactedRow.<init>(PrecompactedRow.java:96)
=0A at
= =0Aorg.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIt= erator.java:138)
=0A at
=0Aorg.apache.cassandra.io.CompactionItera= tor.getReduced(CompactionIterator.java:107)
=0A at
=0Aorg.apache.c= assandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
=0A= at
=0Aorg.apache.cassandra.utils.ReducingIterator.computeNext(Reduci= ngIterator.java:73)
=0A at
=0Acom.google.common.collect.AbstractIt= erator.tryToComputeNext(AbstractIterator.java:136)
=0A at
=0Acom.g= oogle.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)=0A at
=0Aorg.apache.commons.collections.iterators.FilterIterator.s= etNextObject(FilterIterator.java:183)
=0A at
=0Aorg.apache.commons= collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
=0A= at
=0Aorg.apache.cassandra.db.CompactionManager.doCompaction(Compact= ionManager.java:273)
=0A at
=0Aorg.apache.cassandra.db.CompactionM= anager$1.call(CompactionManager.java:109)
=0A at
=0Aorg.apache.cas= sandra.db.CompactionManager$1.call(CompactionManager.java:87)
=0A at ja= va.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
=0A at= java.util.concurrent.FutureTask.run(FutureTask.java:138)
=0A at
=0A= java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
=0A at
=0Ajava.util.concurrent.ThreadPoolExecutor$Worker= run(ThreadPoolExecutor.java:908)
=0A at java.lang.Thread.run(Thread.ja= va:662)
=0A
--Apple-Webmail-86--71250b74-6267-6315-9700-ffab52b71c77-- --Apple-Webmail-42--71250b74-6267-6315-9700-ffab52b71c77--