Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6514D957E for ; Tue, 11 Oct 2011 22:26:16 +0000 (UTC) Received: (qmail 23886 invoked by uid 500); 11 Oct 2011 22:26:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23781 invoked by uid 500); 11 Oct 2011 22:26:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 23773 invoked by uid 99); 11 Oct 2011 22:26:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Oct 2011 22:26:14 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of amartens@ngmoco.com designates 74.125.149.73 as permitted sender) Received: from [74.125.149.73] (HELO na3sys009aog104.obsmtp.com) (74.125.149.73) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 11 Oct 2011 22:26:06 +0000 Received: from mail-vx0-f172.google.com ([209.85.220.172]) (using TLSv1) by na3sys009aob104.postini.com ([74.125.148.12]) with SMTP; Tue, 11 Oct 2011 15:25:46 PDT Received: by vcbfo11 with SMTP id fo11so78686vcb.31 for ; Tue, 11 Oct 2011 15:25:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.33.177 with SMTP id s17mr20773604vdi.130.1318371943961; Tue, 11 Oct 2011 15:25:43 -0700 (PDT) Received: by 10.52.181.99 with HTTP; Tue, 11 Oct 2011 15:25:43 -0700 (PDT) In-Reply-To: References: <1591D974-1C3B-4A4C-9078-E0582633F0FB@thelastpickle.com> Date: Tue, 11 Oct 2011 15:25:43 -0700 Message-ID: Subject: Re: 0.7.9 RejectedExecutionException From: Ashley Martens To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf307ac8b566d2f804af0d63d0 --20cf307ac8b566d2f804af0d63d0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable So we created a script to check if Cassandra is alive and run it every two minutes. Here are some results for today: Tue Oct 11 18:28:09 UTC 2011 - F this Cassandra bullshit... it died again Tue Oct 11 19:00:10 UTC 2011 - F this Cassandra bullshit... it died again Tue Oct 11 19:30:10 UTC 2011 - F this Cassandra bullshit... it died again Tue Oct 11 20:02:10 UTC 2011 - F this Cassandra bullshit... it died again Tue Oct 11 21:34:10 UTC 2011 - F this Cassandra bullshit... it died again Tue Oct 11 22:06:10 UTC 2011 - F this Cassandra bullshit... it died again And here are some of the log tails: INFO [CompactionExecutor:1] 2011-10-11 18:58:14,909 CompactionManager.java (line 395) Compacting [] INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 172) Completed flushing /var/lib/cassandra/data/ system/HintsColumnFamily-f-568-Data.db (60 bytes) INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 157) Writing Memtable-HintsColumnFamily@1493400027(0 bytes, 1 operations) INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 172) Completed flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-569-Data.db (61 bytes) INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 157) Writing Memtable-HintsColumnFamily@1932871300(0 bytes, 1 operations) INFO [FlushWriter:10] 2011-10-11 18:58:15,031 Memtable.java (line 172) Completed flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-570-Data.db (61 bytes) INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147) Deleted /var/lib/cassandra/data/ system/HintsColumnFamily-f-1066 INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1098 INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1040 INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1071 INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,907 SSTable.java (line 147) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1093 INFO [FlushWriter:8] 2011-10-11 20:00:10,701 Memtable.java (line 157) Writing Memtable-HintsColumnFamily@ 1488536311(0 bytes, 1 operations) INFO [CompactionExecutor:1] 2011-10-11 20:00:10,701 CompactionManager.java (line 395) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily-f-1= 687-Data.db'),SSTableReader(path=3D'/var/lib/cassandra/data/system/HintsCol= umnFamily-f-1688-Data.db'),SSTableReader(path=3D'/var/lib/cassandra/data/sy= stem/HintsColumnFamily-f-1689-Data.db'),SSTableReader(path=3D'/var/lib/cass= andra/data/system/HintsColumnFamily-f-1690-Data.db')] INFO [FlushWriter:8] 2011-10-11 20:00:10,741 Memtable.java (line 172) Completed flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-1691-Data.db (61 bytes) INFO [NonPeriodicTasks:1] 2011-10-11 21:33:26,980 SSTable.java (line 147) Deleted /var/lib/cassandra/data/ system/HintsColumnFamily-f-3349 ERROR [Thread-18] 2011-10-11 21:33:31,452 AbstractCassandraDaemon.java (lin= e 132) Fatal exception in thread Thread[Thread-18,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shu= t down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExec= ution(DebuggableThreadPoolExecutor.java:76) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133= 7) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:385= ) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.ja= va:114) ERROR [Thread-19] 2011-10-11 22:04:39,195 AbstractCassandraDaemon.java (lin= e 132) Fatal exception in thread Thread[Thread-19,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shu= t down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExec= ution(DebuggableThreadPoolExecutor.java:76) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133= 7) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:385= ) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.ja= va:114) I'm going to increase the logging level to DEBUG. Other than that I've got to say that Cassandra 0.7.9 is F'ed in some way or another. --20cf307ac8b566d2f804af0d63d0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable So we created a script to check if Cassandra is alive and run it every two = minutes. Here are some results for today:

Tue Oct 11 18:28:09 UTC 20= 11 - F this Cassandra bullshit... it died again
Tue Oct 11 19:00:10 UTC = 2011 - F this Cassandra bullshit... it died again
Tue Oct 11 19:30:10 UTC 2011 - F this Cassandra bullshit... it died againTue Oct 11 20:02:10 UTC 2011 - F this Cassandra bullshit... it died again=
Tue Oct 11 21:34:10 UTC 2011 - F this Cassandra bullshit... it died aga= in
Tue Oct 11 22:06:10 UTC 2011 - F this Cassandra bullshit... it died again

And here are some of the log tails:

=A0INFO [CompactionExe= cutor:1] 2011-10-11 18:58:14,909 CompactionManager.java (line 395) Compacti= ng []
=A0INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 172) C= ompleted flushing /var/lib/cassandra/data/
system/HintsColu= mnFamily-f-568-Data.db (60 bytes)
=A0INFO [FlushWriter:10] 2011-10-11 18:58:14,951 Memtable.java (line 157) W= riting Memtable-HintsColumnFamily@1493400027(0 bytes, 1 operations)
=A0INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 172) C= ompleted flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-569-Da= ta.db (61 bytes)
=A0INFO [FlushWriter:10] 2011-10-11 18:58:14,991 Memtable.java (line 157) W= riting Memtable-HintsColumnFamily@1932871300(0 bytes, 1 operations)
=A0INFO [FlushWriter:10] 2011-10-11 18:58:15,031 Memtable.java (line 172) C= ompleted flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-570-Da= ta.db (61 bytes)

INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 S= STable.java (line 147) Deleted /var/lib/cassandra/data/
system/HintsColumnFamily-f-1066
=A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147= ) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1098
=A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147= ) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1040
=A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,906 SSTable.java (line 147= ) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1071
=A0INFO [NonPeriodicTasks:1] 2011-10-11 19:29:20,907 SSTable.java (line 147= ) Deleted /var/lib/cassandra/data/system/HintsColumnFamily-f-1093

IN= FO [FlushWriter:8] 2011-10-11 20:00:10,701 Memtable.java (line 157) Writing= Memtable-HintsColumnFamily@
1488536311(0 bytes, 1 operations)
=A0INFO [CompactionExecutor:1] 2011-10-11 20:00:10,701 CompactionManager.ja= va (line 395) Compacting [SSTableReader(path=3D'/var/lib/cassandra/data= /system/HintsColumnFamily-f-1687-Data.db'),SSTableReader(path=3D'/v= ar/lib/cassandra/data/system/HintsColumnFamily-f-1688-Data.db'),SSTable= Reader(path=3D'/var/lib/cassandra/data/system/HintsColumnFamily-f-1689-= Data.db'),SSTableReader(path=3D'/var/lib/cassandra/data/system/Hint= sColumnFamily-f-1690-Data.db')]
=A0INFO [FlushWriter:8] 2011-10-11 20:00:10,741 Memtable.java (line 172) Co= mpleted flushing /var/lib/cassandra/data/system/HintsColumnFamily-f-1691-Da= ta.db (61 bytes)

=A0INFO [NonPeriodicTasks:1] 2011-10-11 21:33:26,= 980 SSTable.java (line 147) Deleted /var/lib/cassandra/data/
system/HintsColumnFamily-f-3349
ERROR [Thread-18] 2011-10-11 21:33:31,452 AbstractCassandraDaemon.java=20 (line 132) Fatal exception in thread Thread[Thread-18,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shu= t down
=A0 =A0 =A0 =A0at org.apache.cassandra.concurrent.DebuggableThreadPoolExec= utor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:76)
=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoo= lExecutor.java:816)
=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPo= olExecutor.java:1337)
=A0 =A0 =A0 =A0at org.apache.cassandra.net.MessagingService.receive(Messag= ingService.java:385)
=A0 =A0 =A0 =A0at org.apache.cassandra.net.IncomingTcpConnection.run(Incom= ingTcpConnection.java:114)

ERROR [Thread-19] 2011-10-11 22:04:39,1= 95=20 AbstractCassandraDaemon.java (line 132) Fatal exception in thread=20 Thread[Thread-19,5,main]
java.util.concurrent.RejectedExecutionException= : ThreadPoolExecutor has shut down
=A0 =A0 =A0 =A0at org.apache.cassandra.concurrent.DebuggableThreadPoolExec= utor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:76)
=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoo= lExecutor.java:816)
=A0 =A0 =A0 =A0at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPo= olExecutor.java:1337)
=A0 =A0 =A0 =A0at org.apache.cassandra.net.MessagingService.receive(Messag= ingService.java:385)
=A0 =A0 =A0 =A0at org.apache.cassandra.net.IncomingTcpConnection.run(Incom= ingTcpConnection.java:114)


I'm going to increase the loggi= ng level to DEBUG. Other than that I've got to say that Cassandra 0.7.9= is F'ed in some way or another.
--20cf307ac8b566d2f804af0d63d0--