cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ernst D Schoen-René <er...@peoplebrowsr.com>
Subject can't keep node running.
Date Mon, 29 Aug 2011 16:04:21 GMT
Hi,
   I've written about this before, but I can't find the original thread.

I have one node on my 16 node cassandra cluster (running 8.1) that will 
not stay running.  All nodes were configured the same way, but this one 
kept going down, so I changed these lines in my cassandra-env.sh:

JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=5"
JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=8"
JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=60"

This changed it so that it didn't crash as quickly, but it also never 
used more than 5G of the 8G heap I assign to cassandra.   At the very 
end of its run, the heap climbs suddenly and then the node crashes with 
this error log:

  INFO [ScheduledTasks:1] 2011-08-29 04:56:23,313 StatusLogger.java 
(line 65) AntiEntropyStage                  0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,647 StatusLogger.java 
(line 65) MigrationStage                    0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java 
(line 65) StreamStage                       0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java 
(line 65) MemtablePostFlusher               1         5
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java 
(line 65) FILEUTILS-DELETE-POOL             0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java 
(line 65) FlushWriter                       1         5
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java 
(line 65) MiscStage                         0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java 
(line 65) FlushSorter                       0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java 
(line 65) InternalResponseStage             0         0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java 
(line 65) HintedHandoff                     1         5
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java 
(line 69) CompactionManager               n/a        14
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java 
(line 81) MessagingService                n/a       0,0
  INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java 
(line 85) ColumnFamily                Memtable ops,data  Row cache 
size/cap  Key cache size/cap
  INFO [GossipStage:1] 2011-08-29 04:56:31,647 Gossiper.java (line 606) 
InetAddress /10.28.0.170 is now UP
  INFO [GossipTasks:1] 2011-08-29 04:56:31,647 Gossiper.java (line 620) 
InetAddress /10.28.0.162 is now dead.
  INFO [GossipStage:1] 2011-08-29 04:57:16,441 Gossiper.java (line 606) 
InetAddress /10.28.0.168 is now UP
  INFO [GossipTasks:1] 2011-08-29 04:57:16,442 Gossiper.java (line 620) 
InetAddress /10.28.0.188 is now dead.
  INFO [ScheduledTasks:1] 2011-08-29 04:57:16,439 StatusLogger.java 
(line 88) system.NodeIdInfo                         0,0                 
0/0                 0/0
ERROR [Thread-43] 2011-08-29 04:57:16,438 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-43,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
  INFO [HintedHandoff:1] 2011-08-29 04:57:16,437 
HintedHandOffManager.java (line 320) Started hinted handoff for endpoint 
/10.28.0.178
ERROR [Thread-34] 2011-08-29 04:56:54,282 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-34,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-21] 2011-08-29 04:56:54,279 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-21,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-44] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-44,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-32] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-32,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-28] 2011-08-29 04:56:45,107 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-28,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
  INFO [HintedHandoff:1] 2011-08-29 04:57:31,844 
HintedHandOffManager.java (line 376) Finished hinted handoff of 0 rows 
to endpoint /10.28.0.178
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,844 StatusLogger.java 
(line 88) system.IndexInfo                          0,0                 
0/0                 1/1
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java 
(line 88) system.LocationInfo                       0,0                 
0/0                 1/1
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java 
(line 88) system.Migrations                         0,0                 
0/0                 0/0
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java 
(line 88) system.HintsColumnFamily                  0,0                 
0/0                 0/0
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java 
(line 88) system.Schema                             0,0                 
0/0                 1/1
  INFO [GossipTasks:1] 2011-08-29 04:57:31,836 Gossiper.java (line 620) 
InetAddress /10.28.0.180 is now dead.
ERROR [Thread-27] 2011-08-29 04:57:31,823 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-27,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
  INFO [GossipStage:1] 2011-08-29 04:57:16,442 Gossiper.java (line 606) 
InetAddress /10.28.0.166 is now UP
  INFO [GossipTasks:1] 2011-08-29 04:57:31,848 Gossiper.java (line 620) 
InetAddress /10.28.0.172 is now dead.
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,848 StatusLogger.java 
(line 88) test.analytics                            0,0                 
0/0                 0/0
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java 
(line 88) analytics.analytics                       0,0                 
0/0           9092/9092
  INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java 
(line 88) creds.cred_all_hash              3415,5449198                 
0/0                 0/0
ERROR [Thread-19] 2011-08-29 04:57:43,020 AbstractCassandraDaemon.java 
(line 113) Fatal exception in thread Thread[Thread-19,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has 
shut down
     at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
     at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
     at 
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
     at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
     at 
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
     at 
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
  INFO [ScheduledTasks:1] 2011-08-29 04:57:43,556 StatusLogger.java 
(line 88) creds.boxed_transactions                  0,0                 
0/0                 0/0
  INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620) 
InetAddress /10.28.0.178 is now dead.
  INFO [ScheduledTasks:1] 2011-08-29 04:57:43,558 StatusLogger.java 
(line 88) creds.cred_users               66046,59484126                 
0/0                 6/6
  INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620) 
InetAddress /10.28.0.190 is now dead.
  INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java 
(line 88) creds.cred_all_link              2616,4824235                 
0/0                 0/0
  INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java 
(line 88) creds.cred_all_mention           4713,5517271                 
0/0                 0/0
  INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java 
(line 88) creds.user_ids



Mime
View raw message