Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D2EAA96D2 for ; Thu, 26 Jan 2012 01:04:09 +0000 (UTC) Received: (qmail 16347 invoked by uid 500); 26 Jan 2012 01:04:07 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 16267 invoked by uid 500); 26 Jan 2012 01:04:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 16259 invoked by uid 99); 26 Jan 2012 01:04:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 01:04:06 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a93.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jan 2012 01:04:00 +0000 Received: from homiemail-a93.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a93.g.dreamhost.com (Postfix) with ESMTP id 3BA628405B for ; Wed, 25 Jan 2012 17:03:37 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=JaoR4SeASM cGQK1qV8yZxPyeXxXBDRCMeBGrqsVn09+Dm43XZa6XimtuVtS+FV2LLnTegjFGFn Sr9c2Lt3Qj+cMqPHuaImG/3HLHSJ0yBMXSfnFut9rLFmgOJMN8shYYf+Sbo7dgW5 TflKU1/TWDQjvawQd6e4tgAoaE8IatBMs= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=odg+96Ka+lJYgyNF /30QJaITRqY=; b=rsvQT9JY1tB9hYn5lUTwUqgmvh/dz4IpS6ew6hZcrn4KaXWD 1pVRt59AJGOSA94Kq75k1th6kY+7UuodnNqi6uO/PLSpbhenEJo0OlFn3FoJ2Yld sRsFWvyQqdg3Ek1nIyaTYaVFWOJfMw0WPwZAIRaaAGlxisQd3llRBQO/n1I= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a93.g.dreamhost.com (Postfix) with ESMTPSA id ED8EB84059 for ; Wed, 25 Jan 2012 17:03:35 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_6E044583-A88E-4EBB-9554-50B0724A49B6" Subject: Re: Cannot start cassandra node anymore Date: Thu, 26 Jan 2012 14:03:32 +1300 In-Reply-To: <1327514902074-7224570.post@n2.nabble.com> To: user@cassandra.apache.org References: <1327514902074-7224570.post@n2.nabble.com> Message-Id: <5AD73786-8580-4B60-8725-300646F4C221@thelastpickle.com> X-Mailer: Apple Mail (2.1251.1) --Apple-Mail=_6E044583-A88E-4EBB-9554-50B0724A49B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii There is someone wrong with the way a composite type value was = serialized. The length of a part on disk is not right.=20 As a work around remove the log file, restart and then repair the node.=20= How it got like that is another question. What was the schema change ?=20= Cheers =20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 26/01/2012, at 7:08 AM, huyle wrote: > Hi, >=20 > I am having similar or the same issue on single node environment. I = did > make schema changes recently. Any recommendation how to recover the = server > without loosing data? Here are the last few lines of debug log during = start > up. >=20 > -3881-11e1-ac7f-12313d23ead3:true:4@1326223353559001,])} > DEBUG 18:02:01,057 Reading mutation at 66336992 > DEBUG 18:02:01,058 replaying mutation for > Springpad.696d6167652d7363616c65722d6d657461: > {ColumnFamily(CassandraOrderedQueue > [0,eb321490-3881-11e1-ac7f-12313d23ead3:true:4@1326223353560000,])} > DEBUG 18:02:01,058 Reading mutation at 66337118 > DEBUG 18:02:01,058 replaying mutation for > Springpad.737072696e674d6f64656c44617461626173652d6d657461: > {ColumnFamily(CassandraOrderedQueue > [0,80dc0cd0-3bc0-11e1-83a8-12313d23ead3:false:8@1326223386668000,])} > DEBUG 18:02:01,058 Reading mutation at 66337255 > DEBUG 18:02:01,058 replaying mutation for > = system.38363233616337302d336263302d313165312d303030302d3233666238346464633= 46633: > {ColumnFamily(Schema > = [Avro/Schema:false:2725@1326223386807,Backups:false:431@1326223386807,Spri= ngpad:false:10814@1326223386807,SpringpadGraph:false:2931@1326223386807,])= } > DEBUG 18:02:01,059 Reading mutation at 66354352 > DEBUG 18:02:01,059 replaying mutation for > system.4d6967726174696f6e73204b6579: {ColumnFamily(Migrations > [8623ac70-3bc0-11e1-0000-23fb84ddc4f3:false:23728@1326223386812,])} > DEBUG 18:02:01,059 Reading mutation at 66378184 > DEBUG 18:02:01,059 replaying mutation for > system.4c617374204d6967726174696f6e: {ColumnFamily(Schema [Last > Migration:false:16@1326223386812,])} > DEBUG 18:02:01,059 Reading mutation at 66378302 > INFO 18:02:01,060 Finished reading > /mnt/cassandra/commitlog/CommitLog-1325861435420.log > ERROR 18:02:01,061 Exception encountered during startup > java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:247) > at > = org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCom= positeType.java:57) > at > = org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(A= bstractCompositeType.java:66) > at > = org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCo= mpositeType.java:129) > at org.apache.cassandra.db.Column.getString(Column.java:250) > at > = org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType= .java:137) > at = org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:280) > at = org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241) > at = org.apache.commons.lang.StringUtils.join(StringUtils.java:3073) > at = org.apache.commons.lang.StringUtils.join(StringUtils.java:3133) > at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:301) > at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) > at > = org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassand= raDaemon.java:215) > at > = org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCass= andraDaemon.java:356) > at > = org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107)= > Exception encountered during startup: null >=20 >=20 >=20 > Thanks! >=20 > Huy >=20 >=20 > -- > View this message in context: = http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cannot-st= art-cassandra-node-anymore-tp7150978p7224570.html > Sent from the cassandra-user@incubator.apache.org mailing list archive = at Nabble.com. --Apple-Mail=_6E044583-A88E-4EBB-9554-50B0724A49B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii There = is someone wrong with the way a composite type value was serialized. The = length of a part on disk is not right. 

As a = work around remove the log file, restart and then repair the = node. 

How it got like that is another = question. What was the schema change = ? 

Cheers  

http://www.thelastpickle.com

On 26/01/2012, at 7:08 AM, huyle wrote:

Hi,

I am having similar or the same issue on = single node environment.  I did
make schema changes recently. =  Any recommendation how to recover the server
without loosing = data?  Here are the last few lines of debug log during = start
up.

-3881-11e1-ac7f-12313d23ead3:true:4@1326223353559001,]= )}
DEBUG 18:02:01,057 Reading mutation at 66336992
DEBUG = 18:02:01,058 replaying mutation = for
Springpad.696d6167652d7363616c65722d6d657461:
{ColumnFamily(Cass= andraOrderedQueue
[0,eb321490-3881-11e1-ac7f-12313d23ead3:true:4@132622= 3353560000,])}
DEBUG 18:02:01,058 Reading mutation at = 66337118
DEBUG 18:02:01,058 replaying mutation = for
Springpad.737072696e674d6f64656c44617461626173652d6d657461:
{Col= umnFamily(CassandraOrderedQueue
[0,80dc0cd0-3bc0-11e1-83a8-12313d23ead3= :false:8@1326223386668000,])}
DEBUG 18:02:01,058 Reading mutation at = 66337255
DEBUG 18:02:01,058 replaying mutation = for
system.38363233616337302d336263302d313165312d303030302d323366623834= 646463346633:
{ColumnFamily(Schema
[Avro/Schema:false:2725@132622338= 6807,Backups:false:431@1326223386807,Springpad:false:10814@1326223386807,S= pringpadGraph:false:2931@1326223386807,])}
DEBUG 18:02:01,059 Reading = mutation at 66354352
DEBUG 18:02:01,059 replaying mutation = for
system.4d6967726174696f6e73204b6579: = {ColumnFamily(Migrations
[8623ac70-3bc0-11e1-0000-23fb84ddc4f3:false:23= 728@1326223386812,])}
DEBUG 18:02:01,059 Reading mutation at = 66378184
DEBUG 18:02:01,059 replaying mutation = for
system.4c617374204d6967726174696f6e: {ColumnFamily(Schema = [Last
Migration:false:16@1326223386812,])}
DEBUG 18:02:01,059 = Reading mutation at 66378302
INFO 18:02:01,060 Finished = reading
/mnt/cassandra/commitlog/CommitLog-1325861435420.log
ERROR = 18:02:01,061 Exception encountered during = startup
java.lang.IllegalArgumentException
at = java.nio.Buffer.limit(Buffer.java:247)
= at
org.apache.cassandra.db.marshal.AbstractCompositeType.getByte= s(AbstractCompositeType.java:57)
= at
org.apache.cassandra.db.marshal.AbstractCompositeType.getWith= ShortLength(AbstractCompositeType.java:66)
= at
org.apache.cassandra.db.marshal.AbstractCompositeType.getStri= ng(AbstractCompositeType.java:129)
at = org.apache.cassandra.db.Column.getString(Column.java:250)
= at
org.apache.cassandra.db.marshal.AbstractType.getColumnsString= (AbstractType.java:137)
at = org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:280)
at = org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
at = org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
at = org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:301) = at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) = at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(Ab= stractCassandraDaemon.java:215)
= at
org.apache.cassandra.service.AbstractCassandraDaemon.activate= (AbstractCassandraDaemon.java:356)
= at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDae= mon.java:107)
Exception encountered during startup: = null



Thanks!

Huy


--
View this = message in context: http://cassandr= a-user-incubator-apache-org.3065146.n2.nabble.com/Cannot-start-cassandra-n= ode-anymore-tp7150978p7224570.html
Sent from the cassandra-user@incubat= or.apache.org mailing list archive at Nabble.com.
= --Apple-Mail=_6E044583-A88E-4EBB-9554-50B0724A49B6--