Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1FABB9EA0 for ; Wed, 12 Oct 2011 00:57:53 +0000 (UTC) Received: (qmail 81338 invoked by uid 500); 12 Oct 2011 00:57:51 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81272 invoked by uid 500); 12 Oct 2011 00:57:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81264 invoked by uid 99); 12 Oct 2011 00:57:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Oct 2011 00:57:51 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates 209.85.215.172 as permitted sender) Received: from [209.85.215.172] (HELO mail-ey0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Oct 2011 00:57:46 +0000 Received: by eyg24 with SMTP id 24so196827eyg.31 for ; Tue, 11 Oct 2011 17:57:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=pRI10RWPKa43Y1dv1BCFWu6HQVOOMqzC6KDP/C1mdXw=; b=UWNmP/9CVKnxhH1o2w1InrqQF8v5Ru6uMGcKYAqTB0zXJs0QBJmajzvROK7VmTVxD9 Y/7QM2ah6lEvs5W+5NeW6PIXxUd5PXUEtsi4elguXsafixgoAjcICStTOHc97qSJfIdL 1FMR29MKcxKsNSk4bQ2NUmdOGHFwk3DKbhPBY= Received: by 10.213.9.16 with SMTP id j16mr583042ebj.90.1318381043143; Tue, 11 Oct 2011 17:57:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.17.71 with HTTP; Tue, 11 Oct 2011 17:57:03 -0700 (PDT) In-Reply-To: References: <1591D974-1C3B-4A4C-9078-E0582633F0FB@thelastpickle.com> From: Jonathan Ellis Date: Tue, 11 Oct 2011 19:57:03 -0500 Message-ID: Subject: Re: 0.7.9 RejectedExecutionException To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable grep -i 'killed process' /var/log/messages On Tue, Oct 11, 2011 at 5:25 PM, Ashley Martens wrote= : > So we created a script to check if Cassandra is alive and run it every tw= o > minutes. Here are some results for today: > > Tue Oct 11 18:28:09 UTC 2011 - F this Cassandra bullshit... it died again > Tue Oct 11 19:00:10 UTC 2011 - F this Cassandra bullshit... it died again > Tue Oct 11 19:30:10 UTC 2011 - F this Cassandra bullshit... it died again > Tue Oct 11 20:02:10 UTC 2011 - F this Cassandra bullshit... it died again > Tue Oct 11 21:34:10 UTC 2011 - F this Cassandra bullshit... it died again > Tue Oct 11 22:06:10 UTC 2011 - F this Cassandra bullshit... it died again > > > And here are some of the log tails: > > =A0INFO [CompactionExecutor:1] 2011-10-11 18:58:14,909 CompactionManager.= java > (line 395) Compacting [] > =A0INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 172) > Completed flushing /var/lib/cassandra/data/ > system/HintsColumnFamily-f-568-Data.db (60 bytes) > =A0INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 157) > Writing Memtable-HintsColumnFamily@1493400027(0 bytes, 1 operations) > =A0INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 172) > Completed flushing > /var/lib/cassandra/data/system/HintsColumnFamily-f-569-Data.db (61 bytes) > =A0INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 157) > Writing Memtable-HintsColumnFamily@1932871300(0 bytes, 1 operations) > =A0INFO [FlushWriter:10] 2011-10-11 18:58:15,031 Memtable.java (line 172) > Completed flushing > /var/lib/cassandra/data/system/HintsColumnFamily-f-570-Data.db (61 bytes) > > INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147) > Deleted /var/lib/cassandra/data/ > system/HintsColumnFamily-f-1066 > =A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 1= 47) > Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1098 > =A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 1= 47) > Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1040 > =A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 1= 47) > Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1071 > =A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,907 SSTable.java (line 1= 47) > Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1093 > > INFO [FlushWriter:8] 2011-10-11 20:00:10,701 Memtable.java (line 157) > Writing Memtable-HintsColumnFamily@ > 1488536311(0 bytes, 1 operations) > =A0INFO [CompactionExecutor:1] 2011-10-11 20:00:10,701 CompactionManager.= java > (line 395) Compacting > [SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily-f= -1687-Data.db'),SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsC= olumnFamily-f-1688-Data.db'),SSTableReader(path=3D'/var/lib/cassandra/data/= system/HintsColumnFamily-f-1689-Data.db'),SSTableReader(path=3D'/var/lib/ca= ssandra/data/system/HintsColumnFamily-f-1690-Data.db')] > =A0INFO [FlushWriter:8] 2011-10-11 20:00:10,741 Memtable.java (line 172) > Completed flushing > /var/lib/cassandra/data/system/HintsColumnFamily-f-1691-Data.db (61 bytes= ) > =A0INFO [NonPeriodicTasks:1] 2011-10-11 21:33:26,980 SSTable.java (line 1= 47) > Deleted /var/lib/cassandra/data/ > system/HintsColumnFamily-f-3349 > ERROR [Thread-18] 2011-10-11 21:33:31,452 AbstractCassandraDaemon.java (l= ine > 132) Fatal exception in thread Thread[Thread-18,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:76) > =A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:81= 6) > =A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1= 337) > =A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:3= 85) > =A0 =A0 =A0 =A0at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:114) > ERROR [Thread-19] 2011-10-11 22:04:39,195 AbstractCassandraDaemon.java (l= ine > 132) Fatal exception in thread Thread[Thread-19,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:76) > =A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:81= 6) > =A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1= 337) > =A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:3= 85) > =A0 =A0 =A0 =A0at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:114) > > I'm going to increase the logging level to DEBUG. Other than that I've go= t > to say that Cassandra 0.7.9 is F'ed in some way or another. > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com