incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tamil selvan R.S" <tamil.3...@gmail.com>
Subject Re: Cassandra 1.0.6 nodetool drain gives lots of batch_mutate exceptions
Date Mon, 30 Jul 2012 10:04:54 GMT
Isn't that obvious http://wiki.apache.org/cassandra/NodeTool check drain

On Mon, Jul 30, 2012 at 11:07 AM, Roshan <codevally@gmail.com> wrote:

> Hi
>
> As a part of the Cassandra upgrade to 1.1.2 from 1.0.6, I am running
> *nodetool drain* node by node to empty the commit logs. When draining a
> particular node, that node accepting READ+WRITE request from the clients
> and
> giving below exceptions.
>
> 2012-07-30 23:08:18,169 ERROR [Cassandra$Processor] Internal error
> processing batch_mutate
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut
> down
>         at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>         at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>         at
>
> org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
>         at
>
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
>         at
> org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
>         at
>
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
>         at
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>         at
>
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> 2012-07-30 23:08:18,174 ERROR [AbstractCassandraDaemon] Fatal exception in
> thread Thread[Thread-6,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut
> down
>         at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>         at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>         at
>
> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
>         at
>
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
>         at
>
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
> 2012-07-30 23:08:18,177 ERROR [AbstractCassandraDaemon] Fatal exception in
> thread Thread[Thread-10,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut
> down
>         at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>         at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>         at
>
> org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
>         at
>
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
>         at
>
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
> 2012-07-30 23:08:18,183 ERROR [Cassandra$Processor] Internal error
> processing batch_mutate
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut
> down
>         at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>         at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>         at
>
> org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
>         at
>
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
>         at
> org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
>         at
>
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
>         at
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>         at
>
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> 2012-07-30 23:08:18,194 ERROR [Cassandra$Processor] Internal error
> processing batch_mutate
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
> shut
> down
>         at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
>         at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
>         at
>
> org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
>         at
>
> org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
>         at
> org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
>         at
>
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
>         at
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
>         at
>
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
>         at
>
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>         at
>
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> After several second I noticed that the Cassandra process is still not
> killed and I killed it manually.
>
> Why is this strange behavior? How should I need to do the drain without
> these exceptions?
>
> Thanks
>
> /Roshan
>
>
>
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-1-0-6-nodetool-drain-gives-lots-of-batch-mutate-exceptions-tp7581497.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>

Mime
View raw message