From cassandra-user-return-805-apmail-incubator-cassandra-user-archive=incubator.apache.org@incubator.apache.org Mon Oct 05 22:40:09 2009 Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 82053 invoked from network); 5 Oct 2009 22:40:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Oct 2009 22:40:09 -0000 Received: (qmail 18121 invoked by uid 500); 5 Oct 2009 22:40:09 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 18109 invoked by uid 500); 5 Oct 2009 22:40:09 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 18099 invoked by uid 99); 5 Oct 2009 22:40:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 22:40:09 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.219.205 as permitted sender) Received: from [209.85.219.205] (HELO mail-ew0-f205.google.com) (209.85.219.205) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 22:39:59 +0000 Received: by ewy1 with SMTP id 1so2786874ewy.34 for ; Mon, 05 Oct 2009 15:39:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=4mTmpf70ZumxxIeSjkM5jmseqDi9lJwgQTD2tGssUxk=; b=sMFHGP2427S3SxJlC74isOMWaojadhf+EXtkvE8eL8Ksf5HnKP77KC4S+9wp7lxdIi SnShRPzO5gYV3OYj/NAPO97xZas9wF7xkbxx5+Zq07TBs8GYpADiV1dgMdSchGEgH+xo IvnTCAnoyIqqdGnDPQft3X5XeoW/OubE4eKww= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=JKomyUmEPdSvz+ftN1LfDO+cuKEbH44jboOOl1Zp9hgUtRSx1d4AxKQ/zYgRj2DUyC Jb049/BkSPRvICdIlO6/5BBWRGw7VvoevTuY+t32WJmZK6QbgYwUmUBA4u/sO5jDR63u HwMUknPFXP8tsBZEms5Ma1mrD3oGco7EWhUX4= MIME-Version: 1.0 Received: by 10.216.49.18 with SMTP id w18mr146947web.100.1254782378958; Mon, 05 Oct 2009 15:39:38 -0700 (PDT) In-Reply-To: <35bb42690910051532i58a84e0cr95b7d64a10b39bb@mail.gmail.com> References: <35bb42690910051532i58a84e0cr95b7d64a10b39bb@mail.gmail.com> Date: Mon, 5 Oct 2009 17:39:38 -0500 Message-ID: Subject: Re: Cassandra Fatal Exceptions From: Jonathan Ellis To: cassandra-user@incubator.apache.org, chris@chriswere.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org looks like you ran into a regression. that's the risk you run on trunk. :) Could you open a ticket for this on https://issues.apache.org/jira/browse/CASSANDRA ? -Jonathan On Mon, Oct 5, 2009 at 5:32 PM, Chris Were wrote: > Hi, > I updated from trunk two days ago, but ever since I've been receiving a > series of Fatal Exceptions. My logs show that about an hour after startin= g > Cassandra I receive a=A0RejectedExecutionException and two Assertion > exceptions. From that point on, the log files just fill up with repeated > TimeoutExceptions (each one followed by two Assertion exceptions). I am i= n > no way familar with Cassandra internals so am at a loss to determine what= 's > happening here. > Here is an excerpt of the first 6 exceptions (the last three continue ad > infinitum): > ERROR [FLUSH-TIMER] 2009-10-05 05:04:14,465 CassandraDaemon.java (line > org.apache.cassandra.service.CassandraDaemon) Fatal exception in thread > Thread[FLUSH-TIMER,5,main] > java.util.concurrent.RejectedExecutionException > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thr= eadPoolExecutor.java:1956) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:81= 6) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1= 337) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorServi= ce.java:106) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.j= ava:926) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStor= e.java:367) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.ja= va:415) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.forceFlushIfExpired(ColumnFamil= yStore.java:407) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table$1.run(Table.java:529) > =A0=A0 =A0 =A0 =A0at java.util.TimerThread.mainLoop(Timer.java:534) > =A0=A0 =A0 =A0 =A0at java.util.TimerThread.run(Timer.java:484) > ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574 > DebuggableThreadPoolExecutor.java (line > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Error in > ThreadPoolExecutor > java.lang.AssertionError > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.java:1= 43) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:44= 2) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:626) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(RowMutatio= n.java:205) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:61) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :39) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574 CassandraDaemon.java > (line org.apache.cassandra.service.CassandraDaemon) Fatal exception > =A0in thread Thread[ROW-MUTATION-STAGE:6,5,main] > java.lang.AssertionError > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.java:1= 43) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:44= 2) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:626) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(RowMutatio= n.java:205) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:61) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :39) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > ERROR [pool-1-thread-1] 2009-10-05 05:04:19,525 StorageProxy.java (line > org.apache.cassandra.service.StorageProxy) error writing key > ec9787399d444098819180163b16abc4 > java.util.concurrent.TimeoutException: Operation timed out - received onl= y 0 > responses from =A0. > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHand= ler.java:92) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.jav= a:165) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.jav= a:468) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer= .java:448) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cas= sandra.java:854) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:6= 27) > =A0=A0 =A0 =A0 =A0at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolS= erver.java:253) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784 > DebuggableThreadPoolExecutor.java (line > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Error in > ThreadPoolExecutor > java.lang.AssertionError > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.java:1= 43) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:44= 2) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:626) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(RowMutatio= n.java:205) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:61) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :39) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784 CassandraDaemon.java > (line org.apache.cassandra.service.CassandraDaemon) Fatal exception in > thread Thread[ROW-MUTATION-STAGE:2,5,main] > java.lang.AssertionError > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable.java:1= 43) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:44= 2) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.java:626) > =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(RowMutatio= n.java:205) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHand= ler.java:61) > =A0=A0 =A0 =A0 =A0at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :39) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > =A0=A0 =A0 =A0 =A0at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636) > Cheers, > Chris