Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B15B77B7E for ; Thu, 20 Oct 2011 19:05:31 +0000 (UTC) Received: (qmail 15476 invoked by uid 500); 20 Oct 2011 19:05:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 15389 invoked by uid 500); 20 Oct 2011 19:05:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 15373 invoked by uid 99); 20 Oct 2011 19:05:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 19:05:29 +0000 X-ASF-Spam-Status: No, hits=-0.6 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ramesh25@gmail.com designates 209.85.160.172 as permitted sender) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 19:05:21 +0000 Received: by gyh20 with SMTP id 20so3756596gyh.31 for ; Thu, 20 Oct 2011 12:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=fhYkAVXcC8rljRTZXrLLQdmbdSWJQDfJfXvTHltuBx4=; b=YUCIGoijPb+y8ReSE8gJ3bkKMAfIQsmCqsfgtQaYn03BTVRRN2y+zJRQg7V8deEf+X oKwm31N7Q3LWoAAU08oobh8Xlryxu8YwAU1pzPTD5ijbZ2aTSz0JWmeajFCxaU3LfN/4 G6gTpVoK/t4yt6JGULc56ekvAsTgZ3TmCT/5w= MIME-Version: 1.0 Received: by 10.42.161.70 with SMTP id s6mr20368999icx.40.1319137499868; Thu, 20 Oct 2011 12:04:59 -0700 (PDT) Received: by 10.42.228.199 with HTTP; Thu, 20 Oct 2011 12:04:59 -0700 (PDT) Date: Thu, 20 Oct 2011 14:04:59 -0500 Message-ID: Subject: Cassandra 1.0: Exception in compaction From: Ramesh Natarajan To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org We are running a 8 node cassandra 1.0 cluster. We are seeing this exception quite often. Any idea how to debug this issue? java.lang.IllegalArgumentException: Illegal Capacity: -2 at java.util.ArrayList.(ArrayList.java:110) at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) few lines before this error INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java (line 237) Writing Memtable-participants@1907757288(6777526/153164388 serialized/live bytes, 199339 ops) INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line 488) Discarding obsolete commit log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log) INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java (line 488) Discarding obsolete commit log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log) INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382 bytes) INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java (line 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live bytes, 426 ops) INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes) INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java (line 237) Writing Memtable-msgid@2099219781(3571249/77183008 serialized/live bytes, 109823 ops) INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db (8125165 bytes) INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java (line 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live bytes, 611301 ops) INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db (48225128 bytes) INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line 488) Discarding obsolete commit log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log) INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java (line 488) Discarding obsolete commit log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log) INFO [MutationStage:88] 2011-10-20 10:52:29,905 ColumnFamilyStore.java (line 664) Enqueuing flush of Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007 ops) INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java (line 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live bytes, 5007 ops) INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes) ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:538,1,main] Another one INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java (line 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live bytes, 1111 ops) INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java (line 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db (142584 bytes) INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')] ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:544,1,main] java.lang.AssertionError at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) INFO [MutationStage:89] 2011-10-20 10:52:46,599 ColumnFamilyStore.java (line 664) Enqueuing flush of Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624 ops) INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java (line 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live bytes, 2624 ops) INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java (line 488) Discarding obsolete commit log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log) INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'), SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')] INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552 CompactionTask.java (line 222) Compacted to [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,]. 1,849,016 to 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS. Time: 494ms. ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:539,1,main] java.lang.AssertionError at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:542,1,main] java.lang.IllegalArgumentException: Illegal Capacity: -3 at java.util.ArrayList.(ArrayList.java:110) at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618 CompactionTask.java (line 222) Compacted to [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,]. 73,285,308 to 54,342,740 (~74% of original) bytes for 223,836 keys at 3.594734MBPS. Time: 14,417ms. INFO [MutationStage:75] 2011-10-20 10:52:53,877 ColumnFamilyStore.java (line 664) Enqueuing flush of Memtable-participants@1814149120(143888/3251705 serialized/live bytes, 4232 ops) INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java (line 237) Writing Memtable-participants@1814149120(143888/3251705 serialized/live bytes, 4232 ops) INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'), SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'), SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'), SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')] ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:542,1,main] java.lang.IllegalArgumentException: Illegal Capacity: -2 at java.util.ArrayList.(ArrayList.java:110) at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573) at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787 CompactionTask.java (line 222) Compacted to [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,]. 3,479,848 to 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS. Time: 605ms. INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806 CompactionTask.java (line 222) Compacted to [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,]. 4,467,791 to 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS. Time: 748ms. thanks Ramesh