Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0AD489A16 for ; Tue, 6 Dec 2011 08:17:12 +0000 (UTC) Received: (qmail 20728 invoked by uid 500); 6 Dec 2011 08:17:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 20653 invoked by uid 500); 6 Dec 2011 08:17:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 20627 invoked by uid 99); 6 Dec 2011 08:17:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2011 08:17:04 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sylvain@datastax.com designates 209.85.220.172 as permitted sender) Received: from [209.85.220.172] (HELO mail-vx0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Dec 2011 08:16:57 +0000 Received: by vcbfy13 with SMTP id fy13so5254596vcb.31 for ; Tue, 06 Dec 2011 00:16:36 -0800 (PST) Received: by 10.52.174.46 with SMTP id bp14mr7656847vdc.107.1323159396211; Tue, 06 Dec 2011 00:16:36 -0800 (PST) MIME-Version: 1.0 Received: by 10.52.108.2 with HTTP; Tue, 6 Dec 2011 00:16:15 -0800 (PST) In-Reply-To: References: From: Sylvain Lebresne Date: Tue, 6 Dec 2011 09:16:15 +0100 Message-ID: Subject: Re: AssertionError in hintedhandoff - 1.0.5 To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Do you mind opening a ticket on https://issues.apache.org/jira/browse/CASSANDRA ? Thanks -- Sylvain On Tue, Dec 6, 2011 at 12:52 AM, Ramesh Natarajan wrot= e: > Hi, > > =A0We are running a 8 node cassandra cluster running cassandra 1.0.5. > All our CF use leveled compaction. =A0We ran a test where we did a lot > of inserts for 3 days. After that we started to run tests where some > of the reads could ask for information that was inserted a while back. > =A0In this scenario we are seeing this assertion error in HintedHandoff. > =A0Is this a known issue? > > ERROR [HintedHandoff:3] 2011-12-05 15:42:04,324 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[HintedHandoff:3,1,main] > java.lang.RuntimeException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:330) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:81) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:353) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of > 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerGet(FutureTas= k.java:222) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.get(FutureTask.java:83) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:326) > =A0 =A0 =A0 =A0... 6 more > Caused by: java.lang.AssertionError: originally calculated column size > of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.LazilyCompactedRow.w= rite(LazilyCompactedRow.java:124) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.sstable.SSTableWriter.append(SS= TableWriter.java:160) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.execu= te(CompactionTask.java:158) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$6.= call(CompactionManager.java:275) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTas= k.java:303) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:138= ) > =A0 =A0 =A0 =A0... 3 more > ERROR [HintedHandoff:3] 2011-12-05 15:42:04,333 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[HintedHandoff:3,1,main] > java.lang.RuntimeException: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:34) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.AssertionError: > originally calculated column size of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:330) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.access$100= (HintedHandOffManager.java:81) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager$2.runMayTh= row(HintedHandOffManager.java:353) > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.WrappedRunnable.run(WrappedR= unnable.java:30) > =A0 =A0 =A0 =A0... 3 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.AssertionError: originally calculated column size of > 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerGet(FutureTas= k.java:222) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.get(FutureTask.java:83) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.HintedHandOffManager.deliverHin= tsToEndpoint(HintedHandOffManager.java:326) > =A0 =A0 =A0 =A0... 6 more > Caused by: java.lang.AssertionError: originally calculated column size > of 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.LazilyCompactedRow.w= rite(LazilyCompactedRow.java:124) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.sstable.SSTableWriter.append(SS= TableWriter.java:160) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.execu= te(CompactionTask.java:158) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$6.= call(CompactionManager.java:275) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTas= k.java:303) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:138= ) > =A0 =A0 =A0 =A0... 3 more > ERROR [CompactionExecutor:9931] 2011-12-05 15:42:04,333 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[CompactionExecutor:9931,1,main] > java.lang.AssertionError: originally calculated column size of > 470937164 but now it is 470294247 > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.LazilyCompactedRow.w= rite(LazilyCompactedRow.java:124) > =A0 =A0 =A0 =A0at org.apache.cassandra.io.sstable.SSTableWriter.append(SS= TableWriter.java:160) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionTask.execu= te(CompactionTask.java:158) > =A0 =A0 =A0 =A0at org.apache.cassandra.db.compaction.CompactionManager$6.= call(CompactionManager.java:275) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask$Sync.innerRun(FutureTas= k.java:303) > =A0 =A0 =A0 =A0at java.util.concurrent.FutureTask.run(FutureTask.java:138= ) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(= ThreadPoolExecutor.java:886) > =A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:908) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) > > Attached is the system.log file. > > thanks > Ramesh