Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F40077CDA for ; Mon, 15 Aug 2011 07:26:41 +0000 (UTC) Received: (qmail 98325 invoked by uid 500); 15 Aug 2011 07:26:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 97248 invoked by uid 500); 15 Aug 2011 07:26:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97189 invoked by uid 99); 15 Aug 2011 07:26:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Aug 2011 07:26:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of watcherfr@gmail.com designates 209.85.210.48 as permitted sender) Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Aug 2011 07:26:16 +0000 Received: by pzk34 with SMTP id 34so2466923pzk.21 for ; Mon, 15 Aug 2011 00:25:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1QVblYHpkW1lcFSelWLd1EmMGIZSK+twRIAP5qSd06k=; b=pmD/CggTy4TShPP06+DTUg05vIx1806BKXXv876K9NjN7oCpEWbX1KC34VtjHKHhs2 NwHomqnogToRaWEKOv3prS2QVNasUm0+xWX2Qj9IMJ4qKagwT4FGOcgqfvrMK98238fS 6+jGxXnOl6UspchM0PlS2anX6jvZEEFV2cgKY= MIME-Version: 1.0 Received: by 10.142.52.17 with SMTP id z17mr1701713wfz.143.1313393156313; Mon, 15 Aug 2011 00:25:56 -0700 (PDT) Received: by 10.142.192.14 with HTTP; Mon, 15 Aug 2011 00:25:56 -0700 (PDT) In-Reply-To: <5868E21E-C1E2-4A97-8AFE-4CADB6B61D19@thelastpickle.com> References: <5868E21E-C1E2-4A97-8AFE-4CADB6B61D19@thelastpickle.com> Date: Mon, 15 Aug 2011 09:25:56 +0200 Message-ID: Subject: Re: Merged counter shard with a count != 0 From: Philippe To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cd242e68853ad04aa862c6f --000e0cd242e68853ad04aa862c6f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable > > It looks like the error was thrown during a minor compaction. There shoul= d > be a log line from the CompactionManager before hand that says "Compactin= g=85" > and lists the SSTables it is going to compact. Check that it's from the s= ame > thread, i.e. [CompactionExecutor:6] in the example below > Ok. > With the node stopped, delete / move the SSTable files from the data > directory for the keyspace. They will have the same ColumnFamily-g-XXX.* > Ah, that's what I had missed. I should have looked into the directory. It's obvious, sorry ! > By the way, you may want to do some stress testing with 60 column familie= s > to make sure thing behave as expected. > Yes, I've been running all our traffic through the new cassandra cluster an= d then replaying it on our legacy infrastructure. Been getting a lot of ops experience by doing that ! > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 15 Aug 2011, at 05:28, Philippe wrote: > > Hi I'm getting the following at startup on one of the nodes on my 3 node > cluster with RF=3D3. > I have 6 keyspaces each with 10 column families that contain supercolumns > that contain only counter columns. > > Looking at > http://www.datastax.com/dev/blog/whats-new-in-cassandra-0-8-part-2-counte= rsI see that I am supposed to "remove > all data for that column family". > Does looking at the previous line for the same thread tell me which colum= n > family this is happening to ? > How do I "remove the data" on that node ? > > Thanks > > ERROR [CompactionExecutor:6] 2011-08-14 19:02:55,117 > AbstractCassandraDaemon.java (line 134) Fatal exception in thread > Thread[CompactionExecutor:6,1,main] > java.lang.RuntimeException: Merged counter shard with a count !=3D 0 (lik= ely > due to #2968). You need to restart this node with > -Dcassandra.renew_counter_id=3Dtrue to fix. > at > org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterCon= text.java:633) > at > org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:= 237) > at > org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:= 273) > at > org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldSha= rds(PrecompactedRow.java:67) > at > org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldSha= rds(PrecompactedRow.java:60) > at > org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow= .java:75) > at > org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(C= ompactionController.java:140) > at > org.apache.cassandra.db.compaction.CompactionIterator.getReduced(Compacti= onIterator.java:123) > at > org.apache.cassandra.db.compaction.CompactionIterator.getReduced(Compacti= onIterator.java:43) > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.= java:74) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera= tor.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:= 135) > at > org.apache.commons.collections.iterators.FilterIterator.setNextObject(Fil= terIterator.java:183) > at > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIte= rator.java:94) > at > org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutS= izeEstimation(CompactionManager.java:569) > at > org.apache.cassandra.db.compaction.CompactionManager.doCompaction(Compact= ionManager.java:506) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionMan= ager.java:141) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionMan= ager.java:107) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > at java.lang.Thread.run(Thread.java:662) > > > --000e0cd242e68853ad04aa862c6f Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
It looks like the error was thrown during a minor = compaction. There should be a log line from the CompactionManager before ha= nd that says "Compacting=85" and lists the SSTables it is going t= o compact. Check that it's from the same thread, i.e. [CompactionExecut= or:6] in the example below=A0
Ok.
=A0
With the node stopped, delete = / move the SSTable files from the data directory for the keyspace. They wil= l have the same ColumnFamily-g-XXX.*=A0
Ah, that's what I had missed. I should have loo= ked into the directory. It's obvious, sorry !
=A0
By the way, you may want to do som= e stress testing with 60 column families to make sure thing behave as expec= ted.=A0
Yes, I've been running all our tra= ffic through the new cassandra cluster and then replaying it on our legacy = infrastructure. Been getting a lot of ops experience by doing that !

=A0

Cheers

<= div>
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelast= pickle.com

On 15 Aug 2011, at 05:28, Philippe wrote:

Hi I'm getting the following at startup on one of = the nodes on my 3 node cluster with RF=3D3.
I have 6 keyspaces ea= ch with 10 column families that contain supercolumns that contain only coun= ter columns.

Looking at=A0http://www.= datastax.com/dev/blog/whats-new-in-cassandra-0-8-part-2-counters I see = that I am supposed to "remove a= ll data for that column family".
Does looking at the previous li= ne for the same thread tell me which column family this is happening to ?
How do I "remove the data&= quot; on that node ?
Thanks

ERROR [CompactionExecutor:6] 2011-08-14 19:02:55,117 AbstractCassa= ndraDaemon.java (line 134) Fatal exception in thread Thread[CompactionExecu= tor:6,1,main]
java.lang.RuntimeException: Merged counter shard with a count !=3D 0 (= likely due to #2968). You need to restart this node with -Dcassandra.renew_= counter_id=3Dtrue to fix.
=A0=A0 =A0 =A0 =A0at org.apache.cassand= ra.db.context.CounterContext.removeOldShards(CounterContext.java:633)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.CounterColumn.removeOldSh= ards(CounterColumn.java:237)
=A0=A0 =A0 =A0 =A0at org.apache.cass= andra.db.CounterColumn.removeOldShards(CounterColumn.java:273)
= =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.PrecompactedRow.rem= oveDeletedAndOldShards(PrecompactedRow.java:67)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.PrecompactedRo= w.removeDeletedAndOldShards(PrecompactedRow.java:60)
=A0=A0 =A0 = =A0 =A0at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(P= recompactedRow.java:75)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionCont= roller.getCompactedRow(CompactionController.java:140)
=A0=A0 =A0 = =A0 =A0at org.apache.cassandra.db.compaction.CompactionIterator.getReduced(= CompactionIterator.java:123)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionIter= ator.getReduced(CompactionIterator.java:43)
=A0=A0 =A0 =A0 =A0at = org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.ja= va:74)
=A0=A0 =A0 =A0 =A0at com.google.common.collect.AbstractIterator.tryToComput= eNext(AbstractIterator.java:140)
=A0=A0 =A0 =A0 =A0at com.google.= common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
=A0=A0 =A0 =A0 =A0at org.apache.commons.collections.iterators.FilterItera= tor.setNextObject(FilterIterator.java:183)
=A0=A0 =A0 =A0 =A0at org.apache.commons.collections.iterators.FilterIt= erator.hasNext(FilterIterator.java:94)
=A0=A0 =A0 =A0 =A0at org.a= pache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEsti= mation(CompactionManager.java:569)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionMana= ger.doCompaction(CompactionManager.java:506)
=A0=A0 =A0 =A0 =A0at= org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionMana= ger.java:141)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionMana= ger$1.call(CompactionManager.java:107)
=A0=A0 =A0 =A0 =A0at java.= util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
=A0= =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:138)<= /div>
=A0=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.ru= nTask(ThreadPoolExecutor.java:886)
=A0=A0 =A0 =A0 =A0at java.util= .concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
=A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662)



--000e0cd242e68853ad04aa862c6f--