Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AA5587B7E for ; Sun, 7 Aug 2011 21:28:03 +0000 (UTC) Received: (qmail 82979 invoked by uid 500); 7 Aug 2011 21:28:01 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 82942 invoked by uid 500); 7 Aug 2011 21:28:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 82934 invoked by uid 99); 7 Aug 2011 21:28:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Aug 2011 21:28:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of watcherfr@gmail.com designates 209.85.210.48 as permitted sender) Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Aug 2011 21:27:52 +0000 Received: by pzk34 with SMTP id 34so7300195pzk.21 for ; Sun, 07 Aug 2011 14:27:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=lTRYrlGD7rFDAHvsugPryG9YygcltIrW2l53WKE4CBc=; b=rP+Vm4Gj9G90CjoPPKsHCLlGTvMpHjs3/3iOAMVkZMu3hteiHpn/azv5tVBKPWbVOZ dupkuBXHwULmxdyTR8vihXxtyhp0x7pW9co6+flPM/rk7HTp/8lDZbj1LK7O6liD6kx7 q3v3RSaXe6fxrX0/7E3KtPTy2hQI6SH6IP5dg= MIME-Version: 1.0 Received: by 10.142.186.8 with SMTP id j8mr5021633wff.208.1312752451241; Sun, 07 Aug 2011 14:27:31 -0700 (PDT) Received: by 10.142.174.1 with HTTP; Sun, 7 Aug 2011 14:27:31 -0700 (PDT) Date: Sun, 7 Aug 2011 23:27:31 +0200 Message-ID: Subject: Replicate on write stage errors From: Philippe To: user Content-Type: multipart/alternative; boundary=000e0cd215e088c98404a9f0ffc7 X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd215e088c98404a9f0ffc7 Content-Type: text/plain; charset=ISO-8859-1 hello, I've got new errors showing up in my cassandra log file since I starting testing batch mutates (and it failed). I have done a rolling restart and they are not disappearing. How can I fix this ? What is this really saying about my data and my cluster ? Thanks ERROR [ReplicateOnWriteStage:35] 2011-08-07 23:22:39,147 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReplicateOnWriteStage:35,5,main] java.lang.RuntimeException: java.lang.IllegalArgumentException: ColumnFamily ColumnFamily(PUBLIC_MONTHLY_11 [SuperColumn(310260 [00000000000200:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, 6}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 6}]@1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -378}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -378}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 17, 12124}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, 14097},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, 3737}]@1312752159034!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 17, -917234}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, -1148333},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, -287859}]@1312752159034!-9223372036854775808,]),]) already has modifications in this mutation: ColumnFamily(PUBLIC_MONTHLY_11 [SuperColumn(31026 [00000000000200:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 13, 387}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, 1991},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, 237}]@1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 13, -31485}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, -138769},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, -18209}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, 8}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 8}]@1312752159034!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -708}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -708}]@1312752159034!-9223372036854775808,]),]) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException: ColumnFamily ColumnFamily(PUBLIC_MONTHLY_11 [SuperColumn(310260 [00000000000200:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, 6}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 6}]@1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -378}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -378}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 17, 12124}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, 14097},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, 3737}]@1312752159034!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 17, -917234}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, -1148333},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, -287859}]@1312752159034!-9223372036854775808,]),]) already has modifications in this mutation: ColumnFamily(PUBLIC_MONTHLY_11 [SuperColumn(31026 [00000000000200:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 13, 387}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, 1991},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, 237}]@1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 13, -31485}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, -138769},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, -18209}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, 8}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 8}]@1312752159034!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -708}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -708}]@1312752159034!-9223372036854775808,]),]) at org.apache.cassandra.db.RowMutation.add(RowMutation.java:123) at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:120) at org.apache.cassandra.service.StorageProxy$5$1.runMayThrow(StorageProxy.java:455) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more --000e0cd215e088c98404a9f0ffc7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable hello,
I've got new errors showing up in my cassandra log file sinc= e I starting testing batch mutates (and it failed).
I have done a rolli= ng restart and they are not disappearing.

How can = I fix this ? What is this really saying about my data and my cluster ?
Thanks

ERROR [ReplicateOnWriteStage:35] = 2011-08-07 23:22:39,147 AbstractCassandraDaemon.java (line 139) Fatal excep= tion in thread Thread[ReplicateOnWriteStage:35,5,main]
java.lang.= RuntimeException: java.lang.IllegalArgumentException: ColumnFamily ColumnFa= mily(PUBLIC_MONTHLY_11 [SuperColumn(310260 [00000000000200:false:[{223a4de0= -b5fb-11e0-0000-826f85850cbd, 2, 6}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff,= 2, 6}]@1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-= b5fb-11e0-0000-826f85850cbd, 2, -378}*,{e0db20e0-b5ff-11e0-0000-01687c4831f= f, 2, -378}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a= 4de0-b5fb-11e0-0000-826f85850cbd, 17, 12124}*,{224ceb80-b5fb-11e0-0000-8487= 83ceb9bf, 28, 14097},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, 3737}]@1312= 752159034!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-00= 00-826f85850cbd, 17, -917234}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, -= 1148333},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, -287859}]@1312752159034= !-9223372036854775808,]),]) already has modifications in this mutation: Col= umnFamily(PUBLIC_MONTHLY_11 [SuperColumn(31026 [00000000000200:false:[{223a= 4de0-b5fb-11e0-0000-826f85850cbd, 13, 387}*,{224ceb80-b5fb-11e0-0000-848783= ceb9bf, 8, 1991},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, 237}]@1312752159= 034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826= f85850cbd, 13, -31485}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, -138769},= {e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, -18209}]@1312752159034!-922337203= 6854775808,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, = 8}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 8}]@1312752159034!-9223372036= 854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -= 708}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -708}]@1312752159034!-92233= 72036854775808,]),])
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(Wr= appedRunnable.java:34)
=A0=A0 =A0 =A0 =A0at java.util.concurrent.= ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
= =A0=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:908)
=A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662)
C= aused by: java.lang.IllegalArgumentException: ColumnFamily ColumnFamily(PUB= LIC_MONTHLY_11 [SuperColumn(310260 [00000000000200:false:[{223a4de0-b5fb-11= e0-0000-826f85850cbd, 2, 6}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, 6}]@= 1312752159034!-9223372036854775808,00000000000201:false:[{223a4de0-b5fb-11e= 0-0000-826f85850cbd, 2, -378}*,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -3= 78}]@1312752159034!-9223372036854775808,00000000000300:false:[{223a4de0-b5f= b-11e0-0000-826f85850cbd, 17, 12124}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf= , 28, 14097},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, 3737}]@131275215903= 4!-9223372036854775808,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f8= 5850cbd, 17, -917234}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 28, -1148333}= ,{e0db20e0-b5ff-11e0-0000-01687c4831ff, 15, -287859}]@1312752159034!-922337= 2036854775808,]),]) already has modifications in this mutation: ColumnFamil= y(PUBLIC_MONTHLY_11 [SuperColumn(31026 [00000000000200:false:[{223a4de0-b5f= b-11e0-0000-826f85850cbd, 13, 387}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, = 8, 1991},{e0db20e0-b5ff-11e0-0000-01687c4831ff, 6, 237}]@1312752159034!-922= 3372036854775808,00000000000201:false:[{223a4de0-b5fb-11e0-0000-826f85850cb= d, 13, -31485}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 8, -138769},{e0db20e= 0-b5ff-11e0-0000-01687c4831ff, 6, -18209}]@1312752159034!-92233720368547758= 08,00000000000300:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, 8}*,{e0d= b20e0-b5ff-11e0-0000-01687c4831ff, 2, 8}]@1312752159034!-922337203685477580= 8,00000000000301:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 2, -708}*,{e= 0db20e0-b5ff-11e0-0000-01687c4831ff, 2, -708}]@1312752159034!-9223372036854= 775808,]),])
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.add(RowMutati= on.java:123)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Counter= Mutation.makeReplicationMutation(CounterMutation.java:120)
=A0=A0= =A0 =A0 =A0at org.apache.cassandra.service.StorageProxy$5$1.runMayThrow(St= orageProxy.java:455)
=A0=A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(Wr= appedRunnable.java:30)
=A0=A0 =A0 =A0 =A0... 3 more

--000e0cd215e088c98404a9f0ffc7--