From user-return-20310-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Aug 29 16:04:54 2011 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D6A617740 for ; Mon, 29 Aug 2011 16:04:54 +0000 (UTC) Received: (qmail 32014 invoked by uid 500); 29 Aug 2011 16:04:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 31755 invoked by uid 500); 29 Aug 2011 16:04:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 31746 invoked by uid 99); 29 Aug 2011 16:04:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Aug 2011 16:04:51 +0000 X-ASF-Spam-Status: No, hits=1.1 required=5.0 tests=FROM_12LTRDOM,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Aug 2011 16:04:42 +0000 Received: by gyf3 with SMTP id 3so5749125gyf.31 for ; Mon, 29 Aug 2011 09:04:21 -0700 (PDT) Received: by 10.42.97.73 with SMTP id m9mr5292827icn.126.1314633860704; Mon, 29 Aug 2011 09:04:20 -0700 (PDT) Received: from snakysnake.local ([204.28.127.114]) by mx.google.com with ESMTPS id p11sm2674898ibf.47.2011.08.29.09.04.19 (version=SSLv3 cipher=OTHER); Mon, 29 Aug 2011 09:04:19 -0700 (PDT) Message-ID: <4E5BB885.1080001@peoplebrowsr.com> Date: Mon, 29 Aug 2011 09:04:21 -0700 From: =?ISO-8859-1?Q?Ernst_D_Schoen-Ren=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: can't keep node running. Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I've written about this before, but I can't find the original thread. I have one node on my 16 node cassandra cluster (running 8.1) that will not stay running. All nodes were configured the same way, but this one kept going down, so I changed these lines in my cassandra-env.sh: JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=5" JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=8" JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=60" This changed it so that it didn't crash as quickly, but it also never used more than 5G of the 8G heap I assign to cassandra. At the very end of its run, the heap climbs suddenly and then the node crashes with this error log: INFO [ScheduledTasks:1] 2011-08-29 04:56:23,313 StatusLogger.java (line 65) AntiEntropyStage 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,647 StatusLogger.java (line 65) MigrationStage 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java (line 65) StreamStage 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java (line 65) MemtablePostFlusher 1 5 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java (line 65) FILEUTILS-DELETE-POOL 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java (line 65) FlushWriter 1 5 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java (line 65) MiscStage 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java (line 65) FlushSorter 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java (line 65) InternalResponseStage 0 0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java (line 65) HintedHandoff 1 5 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java (line 69) CompactionManager n/a 14 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java (line 81) MessagingService n/a 0,0 INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java (line 85) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap INFO [GossipStage:1] 2011-08-29 04:56:31,647 Gossiper.java (line 606) InetAddress /10.28.0.170 is now UP INFO [GossipTasks:1] 2011-08-29 04:56:31,647 Gossiper.java (line 620) InetAddress /10.28.0.162 is now dead. INFO [GossipStage:1] 2011-08-29 04:57:16,441 Gossiper.java (line 606) InetAddress /10.28.0.168 is now UP INFO [GossipTasks:1] 2011-08-29 04:57:16,442 Gossiper.java (line 620) InetAddress /10.28.0.188 is now dead. INFO [ScheduledTasks:1] 2011-08-29 04:57:16,439 StatusLogger.java (line 88) system.NodeIdInfo 0,0 0/0 0/0 ERROR [Thread-43] 2011-08-29 04:57:16,438 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-43,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) INFO [HintedHandoff:1] 2011-08-29 04:57:16,437 HintedHandOffManager.java (line 320) Started hinted handoff for endpoint /10.28.0.178 ERROR [Thread-34] 2011-08-29 04:56:54,282 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-34,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) ERROR [Thread-21] 2011-08-29 04:56:54,279 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-21,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) ERROR [Thread-44] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-44,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) ERROR [Thread-32] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-32,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) ERROR [Thread-28] 2011-08-29 04:56:45,107 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-28,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) INFO [HintedHandoff:1] 2011-08-29 04:57:31,844 HintedHandOffManager.java (line 376) Finished hinted handoff of 0 rows to endpoint /10.28.0.178 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,844 StatusLogger.java (line 88) system.IndexInfo 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java (line 88) system.LocationInfo 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java (line 88) system.Migrations 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java (line 88) system.HintsColumnFamily 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java (line 88) system.Schema 0,0 0/0 1/1 INFO [GossipTasks:1] 2011-08-29 04:57:31,836 Gossiper.java (line 620) InetAddress /10.28.0.180 is now dead. ERROR [Thread-27] 2011-08-29 04:57:31,823 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-27,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) INFO [GossipStage:1] 2011-08-29 04:57:16,442 Gossiper.java (line 606) InetAddress /10.28.0.166 is now UP INFO [GossipTasks:1] 2011-08-29 04:57:31,848 Gossiper.java (line 620) InetAddress /10.28.0.172 is now dead. INFO [ScheduledTasks:1] 2011-08-29 04:57:31,848 StatusLogger.java (line 88) test.analytics 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java (line 88) analytics.analytics 0,0 0/0 9092/9092 INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java (line 88) creds.cred_all_hash 3415,5449198 0/0 0/0 ERROR [Thread-19] 2011-08-29 04:57:43,020 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-19,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117) INFO [ScheduledTasks:1] 2011-08-29 04:57:43,556 StatusLogger.java (line 88) creds.boxed_transactions 0,0 0/0 0/0 INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620) InetAddress /10.28.0.178 is now dead. INFO [ScheduledTasks:1] 2011-08-29 04:57:43,558 StatusLogger.java (line 88) creds.cred_users 66046,59484126 0/0 6/6 INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620) InetAddress /10.28.0.190 is now dead. INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java (line 88) creds.cred_all_link 2616,4824235 0/0 0/0 INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java (line 88) creds.cred_all_mention 4713,5517271 0/0 0/0 INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java (line 88) creds.user_ids