cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Eriksson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8065) Mutationstage error, ThreadPoolExecutor has shut down
Date Tue, 07 Oct 2014 14:36:33 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161943#comment-14161943
] 

Marcus Eriksson commented on CASSANDRA-8065:
--------------------------------------------

no, I need to know what happened *before* the exceptions started

> Mutationstage error, ThreadPoolExecutor has shut down
> -----------------------------------------------------
>
>                 Key: CASSANDRA-8065
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8065
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Casssandra Version: 2.0.9
> Total Clusters/Datacenters: 3
> Nodes in each cluster: 4
> Total # of Nodes: 12
> Replication Factor:3 for each data center
> Node Size of each cluster: C3.xlarge in EC2
>            Reporter: Vishal Mehta
>
> 1. Writes running at about 120 qps w/ TTL at row level for 60 minutes (for 3 column families)
> 2. GC set to run at every 10 minutes (so that we claim our space back from tombstones)
> 3. Other settings set as default.
> After some time (1 hour 30 minutes approximately of tests execution) 11 nodes across
datacenters are Down. And write tests fails, with following exception in /var/log/cassandra/system.log.
> {noformat}
> ERROR [MutationStage:18098] 2014-10-04 01:05:36,525 CassandraDaemon.java (line 199) Exception
in thread Thread[MutationStage:18098,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
>         at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
>         at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
>         at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:145)
>         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
>         at org.apache.cassandra.service.StorageProxy.submitHint(StorageProxy.java:893)
>         at org.apache.cassandra.service.StorageProxy.access$300(StorageProxy.java:68)
>         at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:1958)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> WARN [StorageServiceShutdownHook] 2014-10-04 01:05:36,525 StorageProxy.java (line 2031)
Some hints were not written before shutdown.  This is not supposed to happen.  You should
(a) run repair, and (b) file a bug report
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message