cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arya Goudarzi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5119) RejectedExecutionException During Compactions
Date Mon, 18 Feb 2013 04:23:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580402#comment-13580402
] 

Arya Goudarzi commented on CASSANDRA-5119:
------------------------------------------

Can someone confirm it is a none issue? We are still seeing this Exception very often.
                
> RejectedExecutionException During Compactions
> ---------------------------------------------
>
>                 Key: CASSANDRA-5119
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5119
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.6
>         Environment: Sun Java 1.6.0_37 64 Bit
> Ubuntu 12.04
> m1.xlarge EC2 instances
> 12 node cluster with 20Gb data on each node
> Cassandra 1.1.6
>            Reporter: Arya Goudarzi
>
> Hi, I frequently see the following ERROR and stack trace after CompactionTask.java (line
109) has triggered a compaction job (These are automatic minor compactions. We never trigger
major compactions).
> ERROR [CompactionExecutor:881] 2013-01-05 03:40:07,311 AbstractCassandraDaemon.java (line
135) Exception in thread Thread[CompactionExecutor:881,1,RMI Runtime]
> java.util.concurrent.RejectedExecutionException
>         at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
>         at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
>         at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:67)
>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:826)
>         at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:358)
>         at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:330)
>         at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:324)
>         at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:253)
>         at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992)
>         at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200)
>         at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
>         at org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Looking at the code, it seems the thread has been shutdown when it gets to delayedExecute
and isShutdown check would be true resulting a call to reject. It is not clear to me why this
happens. I am guessing the implication would be the old SSTables would not get deleted which
results in disk space usage to keep going up. Restarting the nodes seem to make this go away
for a while. I could not find anything more useful in the logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message