incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ernst D Schoen-René <er...@peoplebrowsr.com>
Subject cassandra unexpected shutdown
Date Tue, 23 Aug 2011 23:08:41 GMT
Hi,
   I'm running a 16-node cassandra cluster, with a reasonably large 
amount of data per node (~1TB).  Nodes have 16G ram, but heap is set to 8G.

The nodes keep stopping with this output in the log.  Any ideas?

ERROR [Thread-85] 2011-08-23 21:00:38,723 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-85,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [ReadStage:568] 2011-08-23 21:00:38,723 
AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
Thread[ReadStage:568,5,main]
java.lang.OutOfMemoryError: Java heap space
  INFO [HintedHandoff:1] 2011-08-23 21:00:38,720 
HintedHandOffManager.java (line 320) Started hinted handoff for endpoint 
/10.28.0.184
  INFO [GossipStage:2] 2011-08-23 21:00:50,751 Gossiper.java (line 606) 
InetAddress /10.29.20.67 is now UP
ERROR [Thread-34] 2011-08-23 21:00:50,525 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-34,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-36] 2011-08-23 21:00:50,518 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-36,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
  INFO [GossipTasks:1] 2011-08-23 21:00:50,466 Gossiper.java (line 620) 
InetAddress /10.29.20.67 is now dead.
  INFO [HintedHandoff:1] 2011-08-23 21:00:50,751 
HintedHandOffManager.java (line 376) Finished hinted handoff of 0 rows 
to endpoint /10.28.0.184
ERROR [Thread-33] 2011-08-23 21:01:05,048 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-33,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-128] 2011-08-23 21:01:05,048 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-128,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
root@cass1:~#


Mime
View raw message