cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Watson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5359) AssertionError in HintedHandoff and CompactionExecutor
Date Fri, 31 May 2013 21:56:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671845#comment-13671845
] 

John Watson commented on CASSANDRA-5359:
----------------------------------------

We have (4) 1.2.5 nodes experiencing this.

They can never compact system/hints so they are constantly retrying to.
                
> AssertionError in HintedHandoff and CompactionExecutor
> ------------------------------------------------------
>
>                 Key: CASSANDRA-5359
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5359
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.2.2
>         Environment: Centos 6.4
> 3 Nodes
> RF 3
> Vnodes
> Murmur3Partitioner
>            Reporter: Joey Imbasciano
>
> Seeing this on 2 of the 3 nodes pretty consistently. The stack traces seem to happen
consistently together, and I believe they are related.
> HintedHandoff
> {noformat}
> ERROR [HintedHandoff:3] 2013-03-18 20:58:58,643 CassandraDaemon.java (line 132) Exception
in thread Thread[HintedHandoff:3,1,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError:
originally calculated column size of 4
> 53516860 but now it is 453517031
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252)
>         at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89)
>         at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally
calculated column size of 453516860 but now
> it is 453517031
>         at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
>         at java.util.concurrent.FutureTask.get(Unknown Source)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402)
>         ... 7 more
> Caused by: java.lang.AssertionError: originally calculated column size of 453516860 but
now it is 453517031
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         ... 3 more
> ERROR [CompactionExecutor:8] 2013-03-18 20:58:58,645 CassandraDaemon.java (line 132)
Exception in thread Thread[CompactionExecutor:8,
> 1,main]
> java.lang.AssertionError: originally calculated column size of 453516860 but now it is
453517031
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}
> CompactionExecutor
> {noformat}
>  INFO [CompactionExecutor:9] 2013-03-18 21:02:54,486 CompactionController.java (line
158) Compacting large row system/hints:bcc4b89f-
> a2ac-4abe-8420-4f5c2462a65b (453517091 bytes) incrementally
> ERROR [CompactionExecutor:9] 2013-03-18 21:03:16,034 CassandraDaemon.java (line 132)
Exception in thread Thread[CompactionExecutor:9,
> 1,main]
> java.lang.AssertionError: originally calculated column size of 453516860 but now it is
453517031
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> ERROR [HintedHandoff:1] 2013-03-18 21:03:16,035 CassandraDaemon.java (line 132) Exception
in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError:
originally calculated column size of 4
> 53516860 but now it is 453517031
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:406)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:252)
>         at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:89)
>         at org.apache.cassandra.db.HintedHandOffManager$4.runMayThrow(HintedHandOffManager.java:459)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally
calculated column size of 453516860 but now
> it is 453517031
>         at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
>         at java.util.concurrent.FutureTask.get(Unknown Source)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:402)
>         ... 7 more
> Caused by: java.lang.AssertionError: originally calculated column size of 453516860 but
now it is 453517031
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:135)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:159)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:422)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         ... 3 more
> {noformat}
> Please let me know any additional info needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message