cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: Merged counter shard with a count != 0
Date Mon, 15 Aug 2011 07:25:56 GMT
>
> It looks like the error was thrown during a minor compaction. There should
> be a log line from the CompactionManager before hand that says "Compacting…"
> and lists the SSTables it is going to compact. Check that it's from the same
> thread, i.e. [CompactionExecutor:6] in the example below
>
Ok.


> With the node stopped, delete / move the SSTable files from the data
> directory for the keyspace. They will have the same ColumnFamily-g-XXX.*
>
Ah, that's what I had missed. I should have looked into the directory. It's
obvious, sorry !


> By the way, you may want to do some stress testing with 60 column families
> to make sure thing behave as expected.
>
Yes, I've been running all our traffic through the new cassandra cluster and
then replaying it on our legacy infrastructure. Been getting a lot of ops
experience by doing that !



>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15 Aug 2011, at 05:28, Philippe wrote:
>
> Hi I'm getting the following at startup on one of the nodes on my 3 node
> cluster with RF=3.
> I have 6 keyspaces each with 10 column families that contain supercolumns
> that contain only counter columns.
>
> Looking at
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-0-8-part-2-countersI see that
I am supposed to "remove
> all data for that column family".
> Does looking at the previous line for the same thread tell me which column
> family this is happening to ?
> How do I "remove the data" on that node ?
>
> Thanks
>
> ERROR [CompactionExecutor:6] 2011-08-14 19:02:55,117
> AbstractCassandraDaemon.java (line 134) Fatal exception in thread
> Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: Merged counter shard with a count != 0 (likely
> due to #2968). You need to restart this node with
> -Dcassandra.renew_counter_id=true to fix.
>         at
> org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:633)
>         at
> org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:237)
>         at
> org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:273)
>         at
> org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:67)
>         at
> org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:60)
>         at
> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:75)
>         at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:140)
>         at
> org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:123)
>         at
> org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:43)
>         at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:74)
>         at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>         at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>         at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>         at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>         at
> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:569)
>         at
> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
>         at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:141)
>         at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:107)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>

Mime
View raw message