cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan <codeva...@gmail.com>
Subject Cassandra 1.0.6 nodetool drain gives lots of batch_mutate exceptions
Date Mon, 30 Jul 2012 05:37:38 GMT
Hi

As a part of the Cassandra upgrade to 1.1.2 from 1.0.6, I am running
*nodetool drain* node by node to empty the commit logs. When draining a
particular node, that node accepting READ+WRITE request from the clients and
giving below exceptions.

2012-07-30 23:08:18,169 ERROR [Cassandra$Processor] Internal error
processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
	at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
	at
org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
	at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
	at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
	at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
	at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
	at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
	at
org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
	at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
	at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
	at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
	at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
2012-07-30 23:08:18,174 ERROR [AbstractCassandraDaemon] Fatal exception in
thread Thread[Thread-6,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
	at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
	at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
	at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
	at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
2012-07-30 23:08:18,177 ERROR [AbstractCassandraDaemon] Fatal exception in
thread Thread[Thread-10,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
	at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
	at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
	at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
	at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
2012-07-30 23:08:18,183 ERROR [Cassandra$Processor] Internal error
processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
	at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
	at
org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
	at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
	at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
	at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
	at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
	at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
	at
org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
	at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
	at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
	at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
	at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
2012-07-30 23:08:18,194 ERROR [Cassandra$Processor] Internal error
processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
	at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
	at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
	at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
	at
org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
	at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
	at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
	at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
	at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
	at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
	at
org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
	at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
	at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
	at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
	at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

After several second I noticed that the Cassandra process is still not
killed and I killed it manually. 

Why is this strange behavior? How should I need to do the drain without
these exceptions?

Thanks

/Roshan
 




--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-1-0-6-nodetool-drain-gives-lots-of-batch-mutate-exceptions-tp7581497.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message