incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Omid Aladini <omidalad...@gmail.com>
Subject Re: Assertions running Cleanup on a 3-node cluster with Cassandra 1.1.4 and LCS
Date Tue, 11 Sep 2012 09:13:47 GMT
Which version of Cassandra has your data been created initially with?

A bug in Cassandra 1.1.2 and earlier could cause out-of-order sstables
and inter-level overlaps in CFs with Leveled Compaction. Your sstables
generated with 1.1.3 and later should not have this issue [1] [2].

In case you have old Leveled-compacted sstables (generated with 1.1.2
or earlier. including 1.0.x) you need to run offline scrub using
Cassandra 1.1.4 or later via /bin/sstablescrub command so it'll fix
out-of-order sstables and inter-level overlaps caused by previous
versions of LCS. You need to take nodes down in order to run offline
scrub.

> After 3 hours the job is done and there are 11390 compaction tasks pending.
> My question: Can these assertions be ignored? Or do I need to worry about
> it?

They can't be ignored since pending compactions elevate the upper
bound on number of disk seeks you need to make to read a row and you
don't get the nice guarantees of leveled compaction.

Cheers,
Omid

[1] https://issues.apache.org/jira/browse/CASSANDRA-4411
[2] https://issues.apache.org/jira/browse/CASSANDRA-4321

On Mon, Sep 10, 2012 at 6:37 PM, Rudolf van der Leeden
<rudolf.vanderleeden@scoreloop.com> wrote:
> Hi,
>
> I'm getting 5 identical assertions while running 'nodetool cleanup' on a
> Cassandra 1.1.4 node with Load=104G and 80m keys.
> From  system.log :
>
> ERROR [CompactionExecutor:576] 2012-09-10 11:25:50,265
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[CompactionExecutor:576,1,main]
> java.lang.AssertionError
>         at
> org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214)
>         at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158)
>         at
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531)
>         at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
>         at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
>         at
> org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at
> org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> After 3 hours the job is done and there are 11390 compaction tasks pending.
> My question: Can these assertions be ignored? Or do I need to worry about
> it?
>
> Thanks for your help and best regards,
> -Rudolf.
>

Mime
View raw message