Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EEA75BE0E for ; Wed, 4 Jan 2012 21:13:12 +0000 (UTC) Received: (qmail 61506 invoked by uid 500); 4 Jan 2012 21:13:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 61454 invoked by uid 500); 4 Jan 2012 21:13:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 61446 invoked by uid 99); 4 Jan 2012 21:13:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 21:13:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a48.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 21:13:04 +0000 Received: from homiemail-a48.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTP id 5DA4D4F8065 for ; Wed, 4 Jan 2012 13:12:26 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=1EgfHKzNBw 1WlVuJlEDUSxSHP5bLJpBlaFNqKhph/pRka6/Ku2oEjzS1TtQjIe4pHdSkQIrDTG gdS5q30XUw/3mzUNXIDLDyaXkfe46e4+Hf7OUGwKfeIx7MEFGFDQbVUdirC9ENXX Aa0ldABnazmx/6uQwhEn9YjBgoLk/jv5w= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=aansvOddf1Y35Os5 CXMxI49eeKU=; b=yqTkbJfyCECIbf1rg+Cu0fOH1fH2fNu4xNZ/Su/hfEN/fcwR cAJbQKW4ZlSdiYM61daUkuLINfnH6qmlJ8Ep1XUIvQHFhvu/JBUQH7j2my0UOgkd yJTC5xu+5Il73hnmjDa1QavhVHuzHbMe2HtKFUbn33F/xl/OBcOpv2rrj0g= Received: from [172.16.1.4] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a48.g.dreamhost.com (Postfix) with ESMTPSA id 9DB4F4F8062 for ; Wed, 4 Jan 2012 13:11:00 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: multipart/alternative; boundary="Apple-Mail=_E540655A-2603-488B-8507-61F0E59F4B2D" Subject: Re: Cannot start cassandra node anymore Date: Thu, 5 Jan 2012 10:10:54 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1251.1) --Apple-Mail=_E540655A-2603-488B-8507-61F0E59F4B2D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 If you have the time turn logging up to DEBUG and start again, it will = log where it failed. Put the logs aside incase there is a bug there.=20 To get things running again: Move the commit log segment out of the = directory and try the restart. Then run a repair from the node.=20 Have you made any recent schema changes ?=20 Hope that helps.=20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/01/2012, at 5:09 AM, Carlo Pires wrote: > Hi, >=20 > I can't start a node of my cluster. Could someone help me to catch the = problem? >=20 > Using: debian with cassandra 1.0.6. >=20 > root@carlo-laptop:/etc/cassandra# cat /var/log/cassandra/output.log=20 > INFO 13:46:00,596 JVM vendor/version: Java HotSpot(TM) 64-Bit Server = VM/1.6.0_26 > INFO 13:46:00,600 Heap size: 1052770304/1052770304 > INFO 13:46:00,601 Classpath: = /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0= -fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/shar= e/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec= -1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandr= a/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhash= map-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandr= a/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1= .4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/= cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/us= r/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthri= ft-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/= lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.= jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/= lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/us= r/share/cassandra/apache-cassandra-1.0.6.jar:/usr/share/cassandra/apache-c= assandra-thrift-1.0.6.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/s= hare/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/s= hare/cassandra/lib/jamm-0.2.5.jar > INFO 13:46:00,617 Unable to link C library. Native methods will be = disabled. > INFO 13:46:00,627 Loading settings from = file:/etc/cassandra/cassandra.yaml > INFO 13:46:00,767 DiskAccessMode 'auto' determined to be mmap, = indexAccessMode is mmap > INFO 13:46:00,783 Global memtable threshold is enabled at 334MB > INFO 13:46:01,086 Opening = /var/lib/cassandra/data/system/Versions-hc-5 (247 bytes) > INFO 13:46:01,125 Opening = /var/lib/cassandra/data/system/Schema-hc-394 (1307316 bytes) > INFO 13:46:01,125 Opening = /var/lib/cassandra/data/system/Schema-hc-396 (14421 bytes) > INFO 13:46:01,145 Opening = /var/lib/cassandra/data/system/Migrations-hc-394 (2285452 bytes) > INFO 13:46:01,148 Opening = /var/lib/cassandra/data/system/Migrations-hc-396 (18582 bytes) > INFO 13:46:01,159 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-13 (415 bytes) > INFO 13:46:01,162 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-15 (163 bytes) > INFO 13:46:01,162 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-14 (80 bytes) > INFO 13:46:01,175 Opening = /var/lib/cassandra/data/system/HintsColumnFamily-hc-7 (13566 bytes) > INFO 13:46:01,183 Opening = /var/lib/cassandra/data/system/HintsColumnFamily-hc-6 (1019 bytes) > INFO 13:46:01,227 Loading schema version = 5c583e90-3631-11e1-0000-b1d70f5561fa > INFO 13:46:01,572 Creating new commitlog segment = /var/lib/cassandra/commitlog/CommitLog-1325691961572.log > INFO 13:46:01,586 Replaying = /var/lib/cassandra/commitlog/CommitLog-1325586865178.log, = /var/lib/cassandra/commitlog/CommitLog-1325625102454.log, = /var/lib/cassandra/commitlog/CommitLog-1325674574711.log, = /var/lib/cassandra/commitlog/CommitLog-1325676430281.log, = /var/lib/cassandra/commitlog/CommitLog-1325676440840.log, = /var/lib/cassandra/commitlog/CommitLog-1325676472468.log, = /var/lib/cassandra/commitlog/CommitLog-1325690114212.log, = /var/lib/cassandra/commitlog/CommitLog-1325691142058.log, = /var/lib/cassandra/commitlog/CommitLog-1325691453774.log, = /var/lib/cassandra/commitlog/CommitLog-1325691528668.log > INFO 13:46:01,657 Finished reading = /var/lib/cassandra/commitlog/CommitLog-1325586865178.log > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) > at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at = org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)= > Caused by: java.lang.IllegalArgumentException > at java.nio.Buffer.limit(Buffer.java:249) > at = org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCom= positeType.java:57) > at = org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(A= bstractCompositeType.java:66) > at = org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractComp= ositeType.java:82) > at = org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractComp= ositeType.java:37) > at = java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compa= reTo(ConcurrentSkipListMap.java:606) > at = java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.jav= a:878) > at = java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListM= ap.java:1893) > at = org.apache.cassandra.db.ThreadSafeSortedColumns.addColumn(ThreadSafeSorted= Columns.java:88) > at = org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCo= ntainer.java:129) > at = org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCo= ntainer.java:124) > at = org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa= milySerializer.java:148) > at = org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa= milySerializer.java:139) > at = org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySer= ializer.java:132) > at = org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowM= utation.java:401) > at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:285) > at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) > at = org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassand= raDaemon.java:196) > at = org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandr= aDaemon.java:238) > ... 5 more > Cannot load daemon > Service exit with a return value of 3 >=20 --Apple-Mail=_E540655A-2603-488B-8507-61F0E59F4B2D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
http://www.thelastpickle.com
On 5/01/2012, at 5:09 AM, Carlo Pires wrote:

Hi,

I= can't start a node of my cluster. Could someone help me to catch the = problem?

Using: debian with cassandra 1.0.6.

root@carlo-laptop:/etc/cassandra# cat = /var/log/cassandra/output.log
 INFO 13:46:00,596 JVM vendor/version: Java HotSpot(TM) 64-Bit = Server VM/1.6.0_26
 INFO 13:46:00,600 Heap size: = 1052770304/1052770304
 INFO 13:46:00,601 Classpath: = /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0= -fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/shar= e/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec= -1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandr= a/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhash= map-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandr= a/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1= .4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/= cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/us= r/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthri= ft-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/= lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.= jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/= lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/us= r/share/cassandra/apache-cassandra-1.0.6.jar:/usr/share/cassandra/apache-c= assandra-thrift-1.0.6.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/s= hare/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/s= hare/cassandra/lib/jamm-0.2.5.jar
 INFO 13:46:00,617 Unable to link C library. Native methods will be = disabled.
 INFO 13:46:00,627 Loading settings from = file:/etc/cassandra/cassandra.yaml
 INFO 13:46:00,767 = DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO 13:46:00,783 Global memtable threshold is enabled at = 334MB
 INFO 13:46:01,086 Opening = /var/lib/cassandra/data/system/Versions-hc-5 (247 bytes)
 INFO = 13:46:01,125 Opening /var/lib/cassandra/data/system/Schema-hc-394 = (1307316 bytes)
 INFO 13:46:01,125 Opening = /var/lib/cassandra/data/system/Schema-hc-396 (14421 bytes)
 INFO = 13:46:01,145 Opening /var/lib/cassandra/data/system/Migrations-hc-394 = (2285452 bytes)
 INFO 13:46:01,148 Opening = /var/lib/cassandra/data/system/Migrations-hc-396 (18582 bytes)
 INFO 13:46:01,159 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-13 (415 = bytes)
 INFO 13:46:01,162 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-15 (163 = bytes)
 INFO 13:46:01,162 Opening = /var/lib/cassandra/data/system/LocationInfo-hc-14 (80 bytes)
 INFO 13:46:01,175 Opening = /var/lib/cassandra/data/system/HintsColumnFamily-hc-7 (13566 = bytes)
 INFO 13:46:01,183 Opening = /var/lib/cassandra/data/system/HintsColumnFamily-hc-6 (1019 = bytes)
 INFO 13:46:01,227 Loading schema version = 5c583e90-3631-11e1-0000-b1d70f5561fa
 INFO 13:46:01,572 Creating new commitlog segment = /var/lib/cassandra/commitlog/CommitLog-1325691961572.log
 INFO = 13:46:01,586 Replaying = /var/lib/cassandra/commitlog/CommitLog-1325586865178.log, = /var/lib/cassandra/commitlog/CommitLog-1325625102454.log, = /var/lib/cassandra/commitlog/CommitLog-1325674574711.log, = /var/lib/cassandra/commitlog/CommitLog-1325676430281.log, = /var/lib/cassandra/commitlog/CommitLog-1325676440840.log, = /var/lib/cassandra/commitlog/CommitLog-1325676472468.log, = /var/lib/cassandra/commitlog/CommitLog-1325690114212.log, = /var/lib/cassandra/commitlog/CommitLog-1325691142058.log, = /var/lib/cassandra/commitlog/CommitLog-1325691453774.log, = /var/lib/cassandra/commitlog/CommitLog-1325691528668.log
 INFO 13:46:01,657 Finished reading = /var/lib/cassandra/commitlog/CommitLog-1325586865178.log
java.lang.refl= ect.InvocationTargetException
    at = sun.reflect.NativeMethodAccessorImpl.invoke0(Native = Method)
    at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39)
    at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25)
    at = java.lang.reflect.Method.invoke(Method.java:597)
    = at = org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)=
Caused by: java.lang.IllegalArgumentException
    at = java.nio.Buffer.limit(Buffer.java:249)
    at = org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCom= positeType.java:57)
    at = org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(A= bstractCompositeType.java:66)
    at = org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractComp= ositeType.java:82)
    at = org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractComp= ositeType.java:37)
    at = java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compa= reTo(ConcurrentSkipListMap.java:606)
    at = java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.jav= a:878)
    at = java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListM= ap.java:1893)
    at = org.apache.cassandra.db.ThreadSafeSortedColumns.addColumn(ThreadSafeSorted= Columns.java:88)
    at = org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCo= ntainer.java:129)
    at = org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnCo= ntainer.java:124)
    at = org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa= milySerializer.java:148)
    at = org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFa= milySerializer.java:139)
    at = org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySer= ializer.java:132)
    at = org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowM= utation.java:401)
    at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:285)    at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172)    at = org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassand= raDaemon.java:196)
    at = org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandr= aDaemon.java:238)
    ... 5 more
Cannot load = daemon
Service exit with a return value of 3


= --Apple-Mail=_E540655A-2603-488B-8507-61F0E59F4B2D--