Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 985E293DC for ; Fri, 21 Oct 2011 14:34:33 +0000 (UTC) Received: (qmail 18582 invoked by uid 500); 21 Oct 2011 14:34:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 18548 invoked by uid 500); 21 Oct 2011 14:34:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 18540 invoked by uid 99); 21 Oct 2011 14:34:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 14:34:31 +0000 X-ASF-Spam-Status: No, hits=-0.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ramesh25@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-yx0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 14:34:26 +0000 Received: by yxj19 with SMTP id 19so4710406yxj.31 for ; Fri, 21 Oct 2011 07:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=mNJxMxL2oGkpHhkl6/lbF5FV14d1aLD0wNiYgaztBfw=; b=HN0kka3uQ5U2m4nUjVt7uiOU3CQADMXgVwSZYChlDxOyU0LrK9Qf66iiqIC5vv36Bj IbR+Sh0p5SI/Mn3OENld9eGc0xyohGXc9FQ+8S+U/eBGznoz4oy9BLm0vUvsfolPQwDm e9gYsRb86nosyU68WMlhxi+rnA1GVwLfRa5zE= MIME-Version: 1.0 Received: by 10.42.161.70 with SMTP id s6mr25391463icx.40.1319207645298; Fri, 21 Oct 2011 07:34:05 -0700 (PDT) Received: by 10.42.228.199 with HTTP; Fri, 21 Oct 2011 07:34:05 -0700 (PDT) In-Reply-To: References: Date: Fri, 21 Oct 2011 09:34:05 -0500 Message-ID: Subject: Re: Cassandra 1.0: Exception in compaction From: Ramesh Natarajan To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable i am using size based compaction ( the default one ). Also this is on linux= . thanks Ramesh On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne wr= ote: > I believe this is the same as > https://issues.apache.org/jira/browse/CASSANDRA-3306. > > The initial reporter only got this exception with leveled compaction, > is it what you are > using too (to help narrow it down)? Also, are you using windows by any ch= ance? > > -- > Sylvain > > > On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan wr= ote: >> We are running a 8 node cassandra 1.0 cluster. We are seeing this >> exception quite often. Any idea how to debug this issue? >> >> >> java.lang.IllegalArgumentException: Illegal Capacity: -2 >> =A0 =A0 =A0 =A0at java.util.ArrayList.(ArrayList.java:110) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.newSSTables(D= ataTracker.java:573) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.replace(DataT= racker.java:546) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replace(DataTracke= r.java:268) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replaceCompactedSS= Tables(DataTracker.java:232) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.replaceCompa= ctedSSTables(ColumnFamilyStore.java:960) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.exec= ute(CompactionTask.java:199) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:131) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:114) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTa= sk.java:303) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:13= 8) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (ThreadPoolExecutor.java:886) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908) >> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> >> >> >> few lines before this error >> >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line >> 237) Writing Memtable-participants@1907757288(6777526/153164388 >> serialized/live bytes, 199339 ops) >> =A0INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line >> 488) Discarding obsolete commit >> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-131911593869= 1.log) >> =A0INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line >> 488) Discarding obsolete commit >> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-131912206195= 6.log) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line >> 273) Completed flushing >> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line >> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live >> bytes, 426 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line >> 273) Completed flushing >> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line >> 237) Writing Memtable-msgid@2099219781(3571249/77183008 >> serialized/live bytes, 109823 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line >> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db >> (8125165 bytes) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line >> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live >> bytes, 611301 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line >> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db >> (48225128 bytes) >> =A0INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line >> 488) Discarding obsolete commit >> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-131912535647= 7.log) >> =A0INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line >> 488) Discarding obsolete commit >> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-131912568335= 1.log) >> =A0INFO [MutationStage:88] 2011-10-20 10:52:29,905 >> ColumnFamilyStore.java (line 664) Enqueuing flush of >> Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007 >> ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line >> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live >> bytes, 5007 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line >> 273) Completed flushing >> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes) >> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132 >> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >> Thread[CompactionExecutor:538,1,main] >> >> >> >> Another one >> >> >> >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line >> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live >> bytes, 1111 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line >> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db >> (142584 bytes) >> =A0INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778 >> CompactionTask.java (line 119) Compacting >> [SSTableReader(path=3D'/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), >> SSTableReader(path=3D'/var/lib/cassandra/data/MSA/uid-h-294-Data.db')] >> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935 >> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >> Thread[CompactionExecutor:544,1,main] >> java.lang.AssertionError >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.newSSTables(D= ataTracker.java:580) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.replace(DataT= racker.java:546) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replace(DataTracke= r.java:268) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replaceCompactedSS= Tables(DataTracker.java:232) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.replaceCompa= ctedSSTables(ColumnFamilyStore.java:960) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.exec= ute(CompactionTask.java:199) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:131) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:114) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTa= sk.java:303) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:13= 8) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (ThreadPoolExecutor.java:886) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908) >> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> =A0INFO [MutationStage:89] 2011-10-20 10:52:46,599 >> ColumnFamilyStore.java (line 664) Enqueuing flush of >> Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624 >> ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line >> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live >> bytes, 2624 ops) >> >> >> >> >> =A0INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line >> 488) Discarding obsolete commit >> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-131912595914= 3.log) >> =A0INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058 >> CompactionTask.java (line 119) Compacting >> [SSTableReader(path=3D'/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), >> SSTableReader(path=3D'/var/lib/cassandra/data/MSA/uid-h-298-Data.db')] >> =A0INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552 >> CompactionTask.java (line 222) Compacted to >> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. =A01,849,016 to >> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS. >> Time: 494ms. >> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901 >> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >> Thread[CompactionExecutor:539,1,main] >> java.lang.AssertionError >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.newSSTables(D= ataTracker.java:580) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.replace(DataT= racker.java:546) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replace(DataTracke= r.java:268) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replaceCompactedSS= Tables(DataTracker.java:232) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.replaceCompa= ctedSSTables(ColumnFamilyStore.java:960) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.exec= ute(CompactionTask.java:199) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:131) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:114) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTa= sk.java:303) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:13= 8) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (ThreadPoolExecutor.java:886) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908) >> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983 >> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >> Thread[CompactionExecutor:542,1,main] >> java.lang.IllegalArgumentException: Illegal Capacity: -3 >> =A0 =A0 =A0 =A0at java.util.ArrayList.(ArrayList.java:110) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.newSSTables(D= ataTracker.java:573) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.replace(DataT= racker.java:546) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replace(DataTracke= r.java:268) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replaceCompactedSS= Tables(DataTracker.java:232) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.replaceCompa= ctedSSTables(ColumnFamilyStore.java:960) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.exec= ute(CompactionTask.java:199) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:131) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:114) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTa= sk.java:303) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:13= 8) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (ThreadPoolExecutor.java:886) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908) >> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> =A0INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618 >> CompactionTask.java (line 222) Compacted to >> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. =A073,285,308 >> to 54,342,740 (~74% of original) bytes for 223,836 keys at >> 3.594734MBPS. =A0Time: 14,417ms. >> =A0INFO [MutationStage:75] 2011-10-20 10:52:53,877 >> ColumnFamilyStore.java (line 664) Enqueuing flush of >> Memtable-participants@1814149120(143888/3251705 serialized/live bytes, >> 4232 ops) >> =A0INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line >> 237) Writing Memtable-participants@1814149120(143888/3251705 >> serialized/live bytes, 4232 ops) >> >> >> >> =A0INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313 >> CompactionTask.java (line 119) Compacting >> [SSTableReader(path=3D'/var/lib/cassandra/data/MSA/participants-h-93-Dat= a.db'), >> SSTableReader(path=3D'/var/lib/cassandra/data/MSA/participants-h-90-Data= .db'), >> SSTableReader(path=3D'/var/lib/cassandra/data/MSA/participants-h-92-Data= .db'), >> SSTableReader(path=3D'/var/lib/cassandra/data/MSA/participants-h-94-Data= .db')] >> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532 >> AbstractCassandraDaemon.java (line 133) Fatal exception in thread >> Thread[CompactionExecutor:542,1,main] >> java.lang.IllegalArgumentException: Illegal Capacity: -2 >> =A0 =A0 =A0 =A0at java.util.ArrayList.(ArrayList.java:110) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.newSSTables(D= ataTracker.java:573) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker$View.replace(DataT= racker.java:546) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replace(DataTracke= r.java:268) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.DataTracker.replaceCompactedSS= Tables(DataTracker.java:232) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.ColumnFamilyStore.replaceCompa= ctedSSTables(ColumnFamilyStore.java:960) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.exec= ute(CompactionTask.java:199) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:131) >> =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$1= .call(CompactionManager.java:114) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTa= sk.java:303) >> =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:13= 8) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (ThreadPoolExecutor.java:886) >> =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908) >> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> =A0INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787 >> CompactionTask.java (line 222) Compacted to >> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. =A03,479,848 to >> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS. >> Time: 605ms. >> =A0INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806 >> CompactionTask.java (line 222) Compacted to >> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. =A04,467,791 to >> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS. >> Time: 748ms. >> >> >> thanks >> Ramesh >> >