cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5318) Exception in CompactionExecutor thread after nodetool drain
Date Fri, 08 Mar 2013 05:06:15 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596831#comment-13596831
] 

Jonathan Ellis commented on CASSANDRA-5318:
-------------------------------------------

You do know that's what drain is supposed to do, right?
                
> Exception in CompactionExecutor thread after nodetool drain
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-5318
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5318
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.2.2
>         Environment: I am running cluster of 6 nodes in amazon(2 nodes per availability
zone in eu-west region) using EC2snith, Cassandra 1.2.2 on ubuntu 12.04, m1.xlarge and 4 x
450 GB of ephemeral volumes stripped together on each node.
>            Reporter: Michael CzerwiƄski
>
> I am doing batch inserts around 5K rows per batch, 400 writes per second.
> I am using leveled compaction and multithreaded_compaction enabled.
> After I do "nodetool drain" I see in the logs memtables being flushed after which I see
exception in CompactionExecutor Thread.
> INFO 15:20:17,471 Completed flushing /mnt/cassandra/data/OpsCenter/pdps/OpsCenter-pdps-ib-60-Data.db
(316475 bytes) for commitlog position ReplayPosition(segmentId=1362668614841, position=26678466)
>  INFO 15:20:17,472 Writing Memtable-rollups300@466350667(305256/9115571 serialized/live
bytes, 12719 ops)
>  INFO 15:20:17,568 Completed flushing /mnt/cassandra/data/OpsCenter/rollups300/OpsCenter-rollups300-ib-60-Data.db
(275536 bytes) for commitlog position ReplayPosition(segmentId=1362668614841, position=26678466)
>  INFO 15:20:17,687 DRAINED
> ERROR 15:20:18,271 Exception in thread Thread[CompactionExecutor:15,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
>         at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
>         at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
>         at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:21,102 Exception in thread Thread[CompactionExecutor:14,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
>         at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
>         at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
>         at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:22,760 Exception in thread Thread[CompactionExecutor:13,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
>         at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
>         at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
>         at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR 15:20:24,587 Exception in thread Thread[CompactionExecutor:16,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
>         at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:63)
>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:902)
>         at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:361)
>         at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:227)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:997)
>         at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:231)
>         at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> This is replicable for me under this load by simply running nodetool drain.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message