From user-return-8382-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Aug 10 00:42:54 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 44345 invoked from network); 10 Aug 2010 00:42:54 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 10 Aug 2010 00:42:54 -0000 Received: (qmail 3184 invoked by uid 500); 10 Aug 2010 00:42:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 3143 invoked by uid 500); 10 Aug 2010 00:42:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 3135 invoked by uid 99); 10 Aug 2010 00:42:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:42:52 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of agoudarzi@gaiaonline.com designates 208.85.94.60 as permitted sender) Received: from [208.85.94.60] (HELO zms-vm01.sv3.gaiaonline.com) (208.85.94.60) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:42:45 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by zms-vm01.sv3.gaiaonline.com (Postfix) with ESMTP id 58CD7CD4002D; Mon, 9 Aug 2010 17:42:24 -0700 (PDT) X-Virus-Scanned: amavisd-new at zms-vm01.sv3.gaiaonline.com Received: from zms-vm01.sv3.gaiaonline.com ([127.0.0.1]) by localhost (zms-vm01.sv3.gaiaonline.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id szrgugxsGfoS; Mon, 9 Aug 2010 17:42:23 -0700 (PDT) Received: from zms-vm01.sv3.gaiaonline.com (zms-vm01.sv3.gaiaonline.com [208.85.94.60]) by zms-vm01.sv3.gaiaonline.com (Postfix) with ESMTP id AFB23CD40026; Mon, 9 Aug 2010 17:42:23 -0700 (PDT) Date: Mon, 9 Aug 2010 17:42:23 -0700 (PDT) From: Arya Goudarzi To: user@cassandra.apache.org Cc: jbellis@gmail.com Message-ID: <3170004.25.1281400942004.JavaMail.arya@aryanet> In-Reply-To: Subject: Re: COMMIT-LOG_WRITER Assertion Error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 6.0.7_GA_2473.RHEL5_64 (Zimbra Desktop/2.0_10479_Mac) X-Virus-Checked: Checked by ClamAV on apache.org I've never run 0.6. I have been running of trunc with automatic svn update = and build everyday at 2pm. One of my nodes got this error which lead to the= same last error prior to build and restart today. Hope this helps better: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.l= ang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.Exec= utionException: java.lang.AssertionError at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities= .java:549) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.ja= va:339) at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.ja= va:174) at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemo= n.java:120) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(Ab= stractCassandraDaemon.java:90) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon= .java:224) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeExcept= ion: java.lang.RuntimeException: java.util.concurrent.ExecutionException: j= ava.lang.AssertionError at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:25= 2) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities= .java:545) ... 5 more Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.uti= l.concurrent.ExecutionException: java.lang.AssertionError at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja= va:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:33= 4) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionExcept= ion: java.lang.AssertionError at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegm= ents(CommitLog.java:408) at org.apache.cassandra.db.ColumnFamilyStore$2.runMayThrow(ColumnFa= milyStore.java:445) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:30) ... 6 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionErro= r at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:25= 2) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegm= ents(CommitLog.java:400) ... 8 more Caused by: java.lang.AssertionError at org.apache.cassandra.db.commitlog.CommitLogHeader$CommitLogHeade= rSerializer.serialize(CommitLogHeader.java:157) at org.apache.cassandra.db.commitlog.CommitLogHeader.writeCommitLog= Header(CommitLogHeader.java:124) at org.apache.cassandra.db.commitlog.CommitLogSegment.writeHeader(C= ommitLogSegment.java:70) at org.apache.cassandra.db.commitlog.CommitLog.discardCompletedSegm= entsInternal(CommitLog.java:450) at org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog= .java:75) at org.apache.cassandra.db.commitlog.CommitLog$6.call(CommitLog.jav= a:394) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:33= 4) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorServi= ce$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.j= ava:30) ... 1 more=20 ----- Original Message ----- From: "Jonathan Ellis" To: user@cassandra.apache.org Sent: Monday, August 9, 2010 5:18:35 PM Subject: Re: COMMIT-LOG_WRITER Assertion Error Sounds like you upgraded to trunk from 0.6 without draining your commitlog first? On Mon, Aug 9, 2010 at 3:30 PM, Arya Goudarzi wr= ote: > Just throwing this out there as it could be a concern. I had a cluster of= 3 nodes running. Over the weekend I updated to trunc (Aug 9th @ 2pm). Toda= y, I came to run my daily tests and my client kept giving me TSocket timeou= ts. Checking the error log of Cassandra servers, all 3 nodes had this and t= hey all became unresponsive! Not sure how to reproduce this but a restart o= f all 3 nodes fixed the issue: > > ERROR [COMMIT-LOG-WRITER] 2010-08-09 11:30:27,722 CassandraDaemon.java (l= ine 82) Uncaught exception in thread Thread[COMMIT-LOG-WRITER,5,main] > java.lang.AssertionError > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.CommitLog= Header$CommitLogHeaderSerializer.serialize(CommitLogHeader.java:157) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.CommitLog= Header.writeCommitLogHeader(CommitLogHeader.java:124) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.CommitLog= Segment.writeHeader(CommitLogSegment.java:70) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.CommitLog= Segment.write(CommitLogSegment.java:103) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.CommitLog= $LogRecordAdder.run(CommitLog.java:521) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.commitlog.PeriodicC= ommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java= :52) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.WrappedRunnable.= run(WrappedRunnable.java:30) > =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:636) > > > -Arya > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com