incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Re: Cassandra 1.0: Exception in compaction
Date Wed, 26 Oct 2011 16:46:49 GMT
No Unfortunately I don't have the log files , the system was scratch loaded. 

Thanks
Ramesh

Sylvain Lebresne <sylvain@datastax.com> wrote:

>You don't have the full logs of a node leading to the exception by any change ?
>Especially one that leads to a
>  java.lang.IllegalArgumentException: Illegal Capacity: -2
>would be great.
>
>--
>Sylvain
>
>On Wed, Oct 26, 2011 at 6:26 PM, Ramesh Natarajan <ramesh25@gmail.com> wrote:
>> We are using the final 1.0.0.
>>
>> Thanks
>> Ramesh
>>
>> On Wed, Oct 26, 2011 at 11:16 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:
>>> Also, to be sure, were you using the 1.0.0 final or some RC when getting this
>>> exception?
>>>
>>> On Fri, Oct 21, 2011 at 8:24 PM, Sylvain Lebresne <sylvain@datastax.com>
wrote:
>>>> Would you have the full log for one of those node leading to the exception
>>>> that you could share? Not sure that'll help but who knows.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>> On Fri, Oct 21, 2011 at 4:34 PM, Ramesh Natarajan <ramesh25@gmail.com>
wrote:
>>>>> i am using size based compaction ( the default one ). Also this is on
linux.
>>>>>
>>>>> thanks
>>>>> Ramesh
>>>>>
>>>>> On Fri, Oct 21, 2011 at 4:25 AM, Sylvain Lebresne <sylvain@datastax.com>
wrote:
>>>>>> I believe this is the same as
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-3306.
>>>>>>
>>>>>> The initial reporter only got this exception with leveled compaction,
>>>>>> is it what you are
>>>>>> using too (to help narrow it down)? Also, are you using windows by
any chance?
>>>>>>
>>>>>> --
>>>>>> Sylvain
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 20, 2011 at 9:04 PM, Ramesh Natarajan <ramesh25@gmail.com>
wrote:
>>>>>>> We are running a 8 node cassandra 1.0 cluster. We are seeing
this
>>>>>>> exception quite often. Any idea how to debug this issue?
>>>>>>>
>>>>>>>
>>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>>>>        at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>>>        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> few lines before this error
>>>>>>>
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:25,885 Memtable.java
(line
>>>>>>> 237) Writing Memtable-participants@1907757288(6777526/153164388
>>>>>>> serialized/live bytes, 199339 ops)
>>>>>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java
(line
>>>>>>> 488) Discarding obsolete commit
>>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319115938691.log)
>>>>>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:25,886 CommitLog.java
(line
>>>>>>> 488) Discarding obsolete commit
>>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319122061956.log)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,865 Memtable.java
(line
>>>>>>> 273) Completed flushing
>>>>>>> /var/lib/cassandra/data/MSA/participants-h-87-Data.db (14695382
bytes)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,866 Memtable.java
(line
>>>>>>> 237) Writing Memtable-modseq@1745889706(13206/311769 serialized/live
>>>>>>> bytes, 426 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,896 Memtable.java
(line
>>>>>>> 273) Completed flushing
>>>>>>> /var/lib/cassandra/data/MSA/modseq-h-262-Data.db (38646 bytes)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:26,897 Memtable.java
(line
>>>>>>> 237) Writing Memtable-msgid@2099219781(3571249/77183008
>>>>>>> serialized/live bytes, 109823 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,497 Memtable.java
(line
>>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/msgid-h-47-Data.db
>>>>>>> (8125165 bytes)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:27,498 Memtable.java
(line
>>>>>>> 237) Writing Memtable-uid@578022704(43734344/317200563 serialized/live
>>>>>>> bytes, 611301 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,802 Memtable.java
(line
>>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-291-Data.db
>>>>>>> (48225128 bytes)
>>>>>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java
(line
>>>>>>> 488) Discarding obsolete commit
>>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125356477.log)
>>>>>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:29,804 CommitLog.java
(line
>>>>>>> 488) Discarding obsolete commit
>>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125683351.log)
>>>>>>>  INFO [MutationStage:88] 2011-10-20 10:52:29,905
>>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>>> Memtable-modseq@339630706(155217/3664394 serialized/live bytes,
5007
>>>>>>> ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:29,905 Memtable.java
(line
>>>>>>> 237) Writing Memtable-modseq@339630706(155217/3664394 serialized/live
>>>>>>> bytes, 5007 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:30,216 Memtable.java
(line
>>>>>>> 273) Completed flushing
>>>>>>> /var/lib/cassandra/data/MSA/modseq-h-263-Data.db (450477 bytes)
>>>>>>> ERROR [CompactionExecutor:538] 2011-10-20 10:52:36,132
>>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>>> Thread[CompactionExecutor:538,1,main]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Another one
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:39,623 Memtable.java
(line
>>>>>>> 237) Writing Memtable-uid@2018688194(79740/578345 serialized/live
>>>>>>> bytes, 1111 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:39,777 Memtable.java
(line
>>>>>>> 273) Completed flushing /var/lib/cassandra/data/MSA/uid-h-295-Data.db
>>>>>>> (142584 bytes)
>>>>>>>  INFO [CompactionExecutor:544] 2011-10-20 10:52:39,778
>>>>>>> CompactionTask.java (line 119) Compacting
>>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-294-Data.db')]
>>>>>>> ERROR [CompactionExecutor:544] 2011-10-20 10:52:39,935
>>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>>> Thread[CompactionExecutor:544,1,main]
>>>>>>> java.lang.AssertionError
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>>>        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>>>  INFO [MutationStage:89] 2011-10-20 10:52:46,599
>>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>>> Memtable-msgid@917568062(85287/1843250 serialized/live bytes,
2624
>>>>>>> ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:46,600 Memtable.java
(line
>>>>>>> 237) Writing Memtable-msgid@917568062(85287/1843250 serialized/live
>>>>>>> bytes, 2624 ops)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  INFO [COMMIT-LOG-WRITER] 2011-10-20 10:52:47,057 CommitLog.java
(line
>>>>>>> 488) Discarding obsolete commit
>>>>>>> log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1319125959143.log)
>>>>>>>  INFO [CompactionExecutor:544] 2011-10-20 10:52:47,058
>>>>>>> CompactionTask.java (line 119) Compacting
>>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-295-Data.db'),
>>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-298-Data.db')]
>>>>>>>  INFO [CompactionExecutor:544] 2011-10-20 10:52:47,552
>>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>>> [/var/lib/cassandra/data/MSA/uid-h-299-Data.db,].  1,849,016
to
>>>>>>> 1,842,872 (~99% of original) bytes for 13,592 keys at 3.557692MBPS.
>>>>>>> Time: 494ms.
>>>>>>> ERROR [CompactionExecutor:539] 2011-10-20 10:52:49,901
>>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>>> Thread[CompactionExecutor:539,1,main]
>>>>>>> java.lang.AssertionError
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:580)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>>>        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:51,983
>>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>>> Thread[CompactionExecutor:542,1,main]
>>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -3
>>>>>>>        at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>>>        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>>>  INFO [CompactionExecutor:540] 2011-10-20 10:52:53,618
>>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>>> [/var/lib/cassandra/data/MSA/participants-h-89-Data.db,].  73,285,308
>>>>>>> to 54,342,740 (~74% of original) bytes for 223,836 keys at
>>>>>>> 3.594734MBPS.  Time: 14,417ms.
>>>>>>>  INFO [MutationStage:75] 2011-10-20 10:52:53,877
>>>>>>> ColumnFamilyStore.java (line 664) Enqueuing flush of
>>>>>>> Memtable-participants@1814149120(143888/3251705 serialized/live
bytes,
>>>>>>> 4232 ops)
>>>>>>>  INFO [FlushWriter:222] 2011-10-20 10:52:53,878 Memtable.java
(line
>>>>>>> 237) Writing Memtable-participants@1814149120(143888/3251705
>>>>>>> serialized/live bytes, 4232 ops)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  INFO [CompactionExecutor:542] 2011-10-20 10:52:54,313
>>>>>>> CompactionTask.java (line 119) Compacting
>>>>>>> [SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-93-Data.db'),
>>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-90-Data.db'),
>>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-92-Data.db'),
>>>>>>> SSTableReader(path='/var/lib/cassandra/data/MSA/participants-h-94-Data.db')]
>>>>>>> ERROR [CompactionExecutor:542] 2011-10-20 10:52:54,532
>>>>>>> AbstractCassandraDaemon.java (line 133) Fatal exception in thread
>>>>>>> Thread[CompactionExecutor:542,1,main]
>>>>>>> java.lang.IllegalArgumentException: Illegal Capacity: -2
>>>>>>>        at java.util.ArrayList.<init>(ArrayList.java:110)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.newSSTables(DataTracker.java:573)
>>>>>>>        at org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:546)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:268)
>>>>>>>        at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:232)
>>>>>>>        at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:960)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:199)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:131)
>>>>>>>        at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
>>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>        at java.lang.Thread.run(Thread.java:662)
>>>>>>>  INFO [CompactionExecutor:539] 2011-10-20 10:52:54,787
>>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>>> [/var/lib/cassandra/data/MSA/uid-h-303-Data.db,].  3,479,848
to
>>>>>>> 3,418,524 (~98% of original) bytes for 24,789 keys at 5.388692MBPS.
>>>>>>> Time: 605ms.
>>>>>>>  INFO [CompactionExecutor:541] 2011-10-20 10:52:54,806
>>>>>>> CompactionTask.java (line 222) Compacted to
>>>>>>> [/var/lib/cassandra/data/MSA/modseq-h-268-Data.db,].  4,467,791
to
>>>>>>> 4,227,199 (~94% of original) bytes for 43,572 keys at 5.389534MBPS.
>>>>>>> Time: 748ms.
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>> Ramesh
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Mime
View raw message