Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 13524 invoked from network); 30 Apr 2010 13:39:31 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 30 Apr 2010 13:39:31 -0000 Received: (qmail 82602 invoked by uid 500); 30 Apr 2010 13:39:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 82584 invoked by uid 500); 30 Apr 2010 13:39:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 82576 invoked by uid 99); 30 Apr 2010 13:39:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 13:39:30 +0000 X-ASF-Spam-Status: No, hits=-0.2 required=10.0 tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-wy0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 13:39:25 +0000 Received: by wyb35 with SMTP id 35so164893wyb.31 for ; Fri, 30 Apr 2010 06:39:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=HB4HvLg2BK7kcAI86ftbGrEs8dmXFrXKXQcXEOzfFjQ=; b=whoCcrZ/rP10bRyPgRx17HLyjBYpJovCrqkIzYfXnsRuKT5woMS8xWzHOnLQWtCqlA O4LK5i4Bg3DUmYJftuqG51WwlDGw4VValOXl6Lzxe4zljmiXXa5QIBICHpVRJoIJVsSk 0Ncws56VlGYMykfLFYy+Ezv+AlzzmB+/E1uo4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=w0TUlwHESVIV98L0iLwEWeRgZI0R3J4huswFMFaQrn5L0PZ4d5fqHmQZjM2Sf1VRb4 sAAeXo4BWBVFLI/N2lADa90H1dTkdf+qnIbJdTh2BzPafo2oreSIy/H61r3dbkXPfacT lhvzmxF5yxsb213hz0BtL8WQhGeo1Kj2MciFA= Received: by 10.216.91.9 with SMTP id g9mr1938815wef.194.1272634742404; Fri, 30 Apr 2010 06:39:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.22.10 with HTTP; Fri, 30 Apr 2010 06:38:42 -0700 (PDT) In-Reply-To: <250A5E75-5E1E-4005-845A-35C15A18FBDF@gmail.com> References: <4BD84522.9030706@androidconsulting.com> <250A5E75-5E1E-4005-845A-35C15A18FBDF@gmail.com> From: Jonathan Ellis Date: Fri, 30 Apr 2010 08:38:42 -0500 Message-ID: Subject: Re: Inserting files to Cassandra timeouts To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable compaction starts but never finishes. are you inserting all these files into the same row? don't do that. On Fri, Apr 30, 2010 at 3:04 AM, Spacejatsi wrote: > I ran again the test, inserting 64 files (15-25MB per file) with 2 thread= s inserting file file at the time. > First 30 files goes relatively fast in, but then it jams, and finally tim= eouts. This tpstats is taken when the first timeout came. > > I also tested to split the files max of 5 mb per file. That was running f= ine about 50% but that also jammed. > Should I be running the inserts to multiple nodes at the same time, not t= o overload one node only? > > 100305-mac17:bin jussi.pori$ ./nodetool -host localhost tpstats > Pool Name =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Active =A0 Pending =A0 = =A0 =A0Completed > FILEUTILS-DELETE-POOL =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A08 > STREAM-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0= 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 > RESPONSE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0= =A0 =A0 =A0 =A0 =A0 =A0 =A00 > ROW-READ-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0= =A0 =A0 =A0 =A0 =A0 =A0 =A00 > LB-OPERATIONS =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0= =A0 =A0 =A0 =A0 =A0 =A0 =A00 > MESSAGE-DESERIALIZER-POOL =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A00 > GMFD =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0= =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 > LB-TARGET =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 > CONSISTENCY-MANAGER =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 > ROW-MUTATION-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A02 =A0 =A0 =A0 =A0 2 =A0= =A0 =A0 =A0 =A0 =A0 31 > MESSAGE-STREAMING-POOL =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A00 > LOAD-BALANCER-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 > FLUSH-SORTER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0= =A0 =A0 =A0 =A0 =A0 =A00 > MEMTABLE-POST-FLUSHER =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A05 > FLUSH-WRITER-POOL =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 0 =A0= =A0 =A0 =A0 =A0 =A0 =A05 > AE-SERVICE-STAGE =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 > > This is the system.log from that test: > > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:34,174 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613414173.log > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:35,481 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613415481.log > =A0INFO [ROW-MUTATION-STAGE:9] 2010-04-30 10:43:36,106 ColumnFamilyStore.= java (line 357) Standard1 has reached its threshold; switching in a fresh M= emtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cassa= ndra-0.6.1/commitlog/CommitLog-1272613415481.log', position=3D18000316) > =A0INFO [ROW-MUTATION-STAGE:9] 2010-04-30 10:43:36,107 ColumnFamilyStore.= java (line 609) Enqueuing flush of Memtable(Standard1)@505410915 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:43:36,108 Memtable.java (line= 148) Writing Memtable(Standard1)@505410915 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:36,862 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613416862.log > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:39,624 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613419624.log > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:42,555 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613422555.log > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:43:42,603 Memtable.java (line= 162) Completed flushing /Users/jussi.pori/Programs/apache-cassandra-0.6.1/= data/Clutters/Standard1-1-Data.db > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:42,650 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613414173.log) > =A0INFO [ROW-MUTATION-STAGE:15] 2010-04-30 10:43:42,655 ColumnFamilyStore= .java (line 357) Standard1 has reached its threshold; switching in a fresh = Memtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cass= andra-0.6.1/commitlog/CommitLog-1272613422555.log', position=3D145) > =A0INFO [ROW-MUTATION-STAGE:15] 2010-04-30 10:43:42,656 ColumnFamilyStore= .java (line 609) Enqueuing flush of Memtable(Standard1)@481059370 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:43:42,657 Memtable.java (line= 148) Writing Memtable(Standard1)@481059370 > =A0INFO [GC inspection] 2010-04-30 10:43:49,760 GCInspector.java (line 11= 0) GC for ParNew: 2390 ms, 58993544 reclaimed leaving 614178960 used; max i= s 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:43:49,913 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613429913.log > =A0INFO [GC inspection] 2010-04-30 10:44:07,039 GCInspector.java (line 11= 0) GC for ParNew: 13666 ms, 60792648 reclaimed leaving 816424624 used; max = is 4108386304 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:44:07,040 Memtable.java (line= 162) Completed flushing /Users/jussi.pori/Programs/apache-cassandra-0.6.1/= data/Clutters/Standard1-2-Data.db > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:44:08,164 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613415481.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:44:08,164 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613416862.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:44:08,169 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613419624.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:44:10,256 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613450256.log > =A0INFO [ROW-MUTATION-STAGE:21] 2010-04-30 10:44:10,564 ColumnFamilyStore= .java (line 357) Standard1 has reached its threshold; switching in a fresh = Memtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cass= andra-0.6.1/commitlog/CommitLog-1272613450256.log', position=3D6000316) > =A0INFO [ROW-MUTATION-STAGE:21] 2010-04-30 10:44:10,565 ColumnFamilyStore= .java (line 609) Enqueuing flush of Memtable(Standard1)@1457398981 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:44:10,565 Memtable.java (line= 148) Writing Memtable(Standard1)@1457398981 > =A0INFO [GC inspection] 2010-04-30 10:44:19,393 GCInspector.java (line 11= 0) GC for ParNew: 8633 ms, 56607240 reclaimed leaving 930530096 used; max i= s 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:45:02,527 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613502527.log > =A0INFO [GC inspection] 2010-04-30 10:45:58,307 GCInspector.java (line 11= 0) GC for ParNew: 49727 ms, 76431056 reclaimed leaving 1102511632 used; max= is 4108386304 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:47:05,944 Memtable.java (line= 162) Completed flushing /Users/jussi.pori/Programs/apache-cassandra-0.6.1/= data/Clutters/Standard1-3-Data.db > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:47:14,264 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613634264.log > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:48:23,571 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613422555.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:48:23,599 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613429913.log) > =A0INFO [ROW-MUTATION-STAGE:26] 2010-04-30 10:48:24,348 ColumnFamilyStore= .java (line 357) Standard1 has reached its threshold; switching in a fresh = Memtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cass= andra-0.6.1/commitlog/CommitLog-1272613634264.log', position=3D18000316) > =A0INFO [ROW-MUTATION-STAGE:26] 2010-04-30 10:48:24,409 ColumnFamilyStore= .java (line 609) Enqueuing flush of Memtable(Standard1)@47327328 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:48:24,409 Memtable.java (line= 148) Writing Memtable(Standard1)@47327328 > =A0INFO [GC inspection] 2010-04-30 10:49:27,238 GCInspector.java (line 11= 0) GC for ParNew: 44522 ms, 80667552 reclaimed leaving 1234718736 used; max= is 4108386304 > =A0INFO [GC inspection] 2010-04-30 10:50:53,913 GCInspector.java (line 11= 0) GC for ParNew: 73534 ms, 83052224 reclaimed leaving 1414752304 used; max= is 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:51:19,800 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613879800.log > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:52:17,371 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272613937371.log > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:52:18,731 Memtable.java (line= 162) Completed flushing /Users/jussi.pori/Programs/apache-cassandra-0.6.1/= data/Clutters/Standard1-4-Data.db > =A0INFO [COMPACTION-POOL:1] 2010-04-30 10:52:18,880 CompactionManager.jav= a (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path=3D'/Use= rs/jussi.pori/Programs/apache-cassandra-0.6.1/data/Clutters/Standard1-1-Dat= a.db'),org.apache.cassandra.io.SSTableReader(path=3D'/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/data/Clutters/Standard1-2-Data.db'),org.apache.= cassandra.io.SSTableReader(path=3D'/Users/jussi.pori/Programs/apache-cassan= dra-0.6.1/data/Clutters/Standard1-3-Data.db'),org.apache.cassandra.io.SSTab= leReader(path=3D'/Users/jussi.pori/Programs/apache-cassandra-0.6.1/data/Clu= tters/Standard1-4-Data.db')] > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:52:32,070 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613450256.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:52:32,188 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613502527.log) > > > =A0INFO [GC inspection] 2010-04-30 10:53:46,949 GCInspector.java (line 11= 0) GC for ParNew: 13621 ms, 88262512 reclaimed leaving 1581477264 used; max= is 4108386304 > =A0INFO [GC inspection] 2010-04-30 10:55:08,644 GCInspector.java (line 11= 0) GC for ParNew: 15073 ms, 20452712 reclaimed leaving 431259232 used; max = is 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:55:08,644 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272614108644.log > =A0INFO [ROW-MUTATION-STAGE:32] 2010-04-30 10:55:12,952 ColumnFamilyStore= .java (line 357) Standard1 has reached its threshold; switching in a fresh = Memtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cass= andra-0.6.1/commitlog/CommitLog-1272614108644.log', position=3D18000316) > =A0INFO [ROW-MUTATION-STAGE:32] 2010-04-30 10:55:12,953 ColumnFamilyStore= .java (line 609) Enqueuing flush of Memtable(Standard1)@1990541648 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:55:12,967 Memtable.java (line= 148) Writing Memtable(Standard1)@1990541648 > =A0INFO [GC inspection] 2010-04-30 10:55:20,718 GCInspector.java (line 11= 0) GC for ParNew: 6685 ms, 31187112 reclaimed leaving 626981312 used; max i= s 4108386304 > =A0INFO [GC inspection] 2010-04-30 10:55:31,084 GCInspector.java (line 11= 0) GC for ParNew: 10109 ms, 3237800 reclaimed leaving 752990680 used; max i= s 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:55:33,247 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272614133247.log > =A0INFO [GC inspection] 2010-04-30 10:55:41,083 GCInspector.java (line 11= 0) GC for ParNew: 1426 ms, 48780568 reclaimed leaving 1053980248 used; max = is 4108386304 > =A0INFO [GC inspection] 2010-04-30 10:56:01,084 GCInspector.java (line 11= 0) GC for ParNew: 269 ms, 200003712 reclaimed leaving 1500784512 used; max = is 4108386304 > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:56:03,609 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272614163609.log > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:56:13,219 Memtable.java (line= 162) Completed flushing /Users/jussi.pori/Programs/apache-cassandra-0.6.1/= data/Clutters/Standard1-5-Data.db > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:56:13,405 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613634264.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:56:13,406 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613879800.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:56:13,407 CommitLog.java (line = 407) Discarding obsolete commit log:CommitLogSegment(/Users/jussi.pori/Prog= rams/apache-cassandra-0.6.1/commitlog/CommitLog-1272613937371.log) > =A0INFO [COMMIT-LOG-WRITER] 2010-04-30 10:56:20,275 CommitLogSegment.java= (line 50) Creating new commitlog segment /Users/jussi.pori/Programs/apache= -cassandra-0.6.1/commitlog/CommitLog-1272614180275.log > =A0INFO [ROW-MUTATION-STAGE:6] 2010-04-30 10:56:20,300 ColumnFamilyStore.= java (line 357) Standard1 has reached its threshold; switching in a fresh M= emtable at CommitLogContext(file=3D'/Users/jussi.pori/Programs/apache-cassa= ndra-0.6.1/commitlog/CommitLog-1272614180275.log', position=3D145) > =A0INFO [ROW-MUTATION-STAGE:6] 2010-04-30 10:56:20,300 ColumnFamilyStore.= java (line 609) Enqueuing flush of Memtable(Standard1)@1034225086 > =A0INFO [FLUSH-WRITER-POOL:1] 2010-04-30 10:56:20,301 Memtable.java (line= 148) Writing Memtable(Standard1)@1034225086 > =A0INFO [GC inspection] 2010-04-30 10:56:42,053 GCInspector.java (line 11= 0) GC for ParNew: 21602 ms, 58548344 reclaimed leaving 1670216000 used; max= is 4108386304 > =A0INFO [GC inspection] 2010-04-30 10:57:12,785 GCInspector.java (line 11= 0) GC for ParNew: 30465 ms, 3229848 reclaimed leaving 1922229272 used; max = is 4108386304 > > > On Apr 29, 2010, at 3:50 PM, Jonathan Ellis wrote: > >> are you seeing memtable flushes and compactions in the log? >> >> what does tpstats look like when it's timing out? >> >> spending 2000ms on GC every 50s indicates that it's not GC causing >> your problem. =A0(especially when all of them are ParNew, which are >> completely non-blocking to other threads) >> >> On Wed, Apr 28, 2010 at 9:24 AM, Jussi P?=F6ri >> wrote: >>> new try, previous went to wrong place... >>> >>> Hi all, >>> >>> i'm trying to run a scenario of adding files from specific folder to ca= ssandra. Now I have 64 files(about 15-20 MB per file) and overall of 1GB of= data. >>> I'm able to insert a round 40 files, but after that the cassandra goes = to some GC loop and I finally get an timeout to the client. >>> It is not going to OOM, but it just jams. >>> >>> Here is what I had last marks in log file: >>> NFO [GC inspection] 2010-04-28 10:07:55,297 GCInspector.java (line 110)= GC for ParNew: 232 ms, 25731128 reclaimed leaving 553241120 used; max is 4= 108386304 >>> =A0INFO [GC inspection] 2010-04-28 10:09:02,331 GCInspector.java (line = 110) GC for ParNew: 2844 ms, 238909856 reclaimed leaving 1435582832 used; m= ax is 4108386304 >>> =A0INFO [GC inspection] 2010-04-28 10:09:49,421 GCInspector.java (line = 110) GC for ParNew: 30666 ms, 11185824 reclaimed leaving 1679795336 used; m= ax is 4108386304 >>> =A0INFO [GC inspection] 2010-04-28 10:11:18,090 GCInspector.java (line = 110) GC for ParNew: 895 ms, 17921680 reclaimed leaving 1589308456 used; max= is 4108386304 >>> >>> >>> >>> I think that I must have something wrong in my configurations or in how= I use cassandra, because here people are inserting 10 times more stuff and= it works. >>> >>> Column family I using: >>> >>> Basically inserting with key name is "Folder_name" and column name is "= file name" and value is the file content. >>> I tried with Hector(mainly) and directly using thrift(insert and batch_= mutate). >>> >>> In my case, the data does not need to readable immediately after insert= , but I don't know it that helps in anyway. >>> >>> >>> My environment : >>> mac and/or linux, tested in both >>> java 1.6.0_17 >>> Cassandra 0.6.1 >>> >>> >>> >>> =A060000 >>> 32 >>> 512 >>> =A032 >>> =A032 >>> =A08 >>> =A064 >>> =A064 >>> =A0256 >>> =A00.1 >>> =A060 >>> =A08 >>> =A032 >>> =A0batch >>> =A0 >>> =A01.0 >>> =A0500 >>> >>> JVM_OPTS=3D" \ >>> =A0 =A0 =A0 =A0-server \ >>> =A0 =A0 =A0 =A0-Xms3G \ >>> =A0 =A0 =A0 =A0-Xmx3G \ >>> =A0 =A0 =A0 =A0-XX:PermSize=3D512m \ >>> =A0 =A0 =A0 =A0-XX:MaxPermSize=3D800m \ >>> =A0 =A0 =A0 =A0-XX:MaxNewSize=3D256m \ >>> =A0 =A0 =A0 =A0-XX:NewSize=3D128m \ >>> =A0 =A0 =A0 =A0-XX:TargetSurvivorRatio=3D90 \ >>> =A0 =A0 =A0 =A0-XX:+AggressiveOpts \ >>> =A0 =A0 =A0 =A0-XX:+UseParNewGC \ >>> =A0 =A0 =A0 =A0-XX:+UseConcMarkSweepGC \ >>> =A0 =A0 =A0 =A0-XX:+CMSParallelRemarkEnabled \ >>> =A0 =A0 =A0 =A0-XX:+HeapDumpOnOutOfMemoryError \ >>> =A0 =A0 =A0 =A0-XX:SurvivorRatio=3D128 \ >>> =A0 =A0 =A0 =A0-XX:MaxTenuringThreshold=3D0 \ >>> =A0 =A0 =A0 =A0-XX:+DisableExplicitGC \ >>> =A0 =A0 =A0 =A0-Dcom.sun.management.jmxremote.port=3D8080 \ >>> =A0 =A0 =A0 =A0-Dcom.sun.management.jmxremote.ssl=3Dfalse \ >>> =A0 =A0 =A0 =A0-Dcom.sun.management.jmxremote.authenticate=3Dfalse" >>> >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com