Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1D85526DC for ; Fri, 29 Apr 2011 21:02:50 +0000 (UTC) Received: (qmail 94861 invoked by uid 500); 29 Apr 2011 21:02:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94833 invoked by uid 500); 29 Apr 2011 21:02:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94825 invoked by uid 99); 29 Apr 2011 21:02:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 21:02:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a53.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 21:02:40 +0000 Received: from homiemail-a53.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTP id 01BDC138091 for ; Fri, 29 Apr 2011 14:02:17 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=I67A8XdVg0 TtS2KT/bpeMLYnNodPALvLI9nASOG9r9vNJrXufbNi/RiHSHz6p/nWqruhMoWGjm Rc8278CK9bu8DoJB/yrcWuna8PVh6i/ahU1EMgAyowmnrLcRl65Fm2pGovvtPM6w /hLecG/tv7ahsfENvarPwgzWYQSAYwm4I= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=CpWkWgmDt5x1AERA Rp8gA2JKvsM=; b=KQplSCRe3ogP4AOB18QF8Tuv1bbD/s6wH9Uu291WHSlZKZVh qd5dH0LaUpy1CFT/NewGy7XQQW556VxSQ+m5PKxsQB1xLtDVbz2bQlcxc8/xtPMw l2nwCa/aV4siBTjnczDnTkV9TMZgMGWfRuQYyzuWBBWOrb0oSEpv9qYvU1U= Received: from [10.0.1.155] (121-73-157-230.cable.telstraclear.net [121.73.157.230]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a53.g.dreamhost.com (Postfix) with ESMTPSA id 36D4613808B for ; Fri, 29 Apr 2011 14:02:16 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-4--613782371 Subject: Re: Cassandra node throws NPE on startup Date: Sat, 30 Apr 2011 09:02:11 +1200 In-Reply-To: To: user@cassandra.apache.org References: <68416572-6342-4570-8439-9860A158C981@zfabrik.de> <2C6FC182-4F49-48EE-BBB9-F4FA67C4B3D3@thelastpickle.com> <3118DF5B-C12E-428B-A582-B7568D5F8E89@zfabrik.de> Message-Id: X-Mailer: Apple Mail (2.1084) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-4--613782371 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I've not heard of it happening in a production system.=20 The commit log and the schema have diverged, that does not normally = happen.=20 Aaron =20 On 29 Apr 2011, at 20:54, Subscriber wrote: > But isn't their a better way than to delete the commitlogs? > What do I do when this happens in a productive system? >=20 > Ciao > Udo >=20 > Am 29.04.2011 um 02:07 schrieb aaron morton: >=20 >> Thought you may have re-created the schema.=20 >>=20 >> Kill the process like that should be ok, let us know if you get the = error again.=20 >>=20 >> Aaron >>=20 >> On 29 Apr 2011, at 02:56, Subscriber wrote: >>=20 >>> Hi Aaron,=20 >>>=20 >>> what exactly do you mean? >>> I restarted the cluster by calling >>>=20 >>> > bin/cassandra -p pid.file >>>=20 >>> on all three nodes. The first node is the (only) seed. >>>=20 >>> Udo >>>=20 >>>=20 >>> Am 27.04.2011 um 23:28 schrieb Aaron Morton: >>>=20 >>>> What approach did you take to restarting the cluster?=20 >>>>=20 >>>> It looks like the keyspace name was changed and the log replay = tried to write to the old one. >>>>=20 >>>> Aaron >>>>=20 >>>> On 28/04/2011, at 12:03 AM, Subscriber = wrote: >>>>=20 >>>>> Hi again,=20 >>>>>=20 >>>>> some more remarks.=20 >>>>> I renamed the commitlog directory on the third node so that = cassandra cannot see it on startup.=20 >>>>> Now the node starts fine. The problem seems to have something to = do with the commitlogs... >>>>>=20 >>>>> Best Regards >>>>> Udo >>>>>=20 >>>>>=20 >>>>> Am 27.04.2011 um 13:22 schrieb Subscriber: >>>>>=20 >>>>>> Hi,=20 >>>>>>=20 >>>>>> I'm using Cassandra 0.7.4 on a three node cluster. The cluster = was setup yesterday as a fresh installation (no upgrade). >>>>>> The cluster is installed beside a hadoop cluster (I want to = discover how cassandra works together with hadoop's map/reduce feature). >>>>>>=20 >>>>>> After loading some test data into the cassandra cluster (some = text-files for a first wordcount m/r example ;-) I killed all nodes and = restarted the cluster. >>>>>> Two nodes are starting fine but the third node doesn't start and = throws a NullPointerException: >>>>>>=20 >>>>>> INFO 12:44:42,557 Finished reading = /home/.../cassandra/work/commitlog/CommitLog-1303828029742.log >>>>>> ERROR 12:44:42,558 Exception encountered during startup. >>>>>> java.lang.NullPointerException >>>>>> at = org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275) >>>>>> at org.apache.cassandra.db.Table.(Table.java:221) >>>>>> at org.apache.cassandra.db.Table.open(Table.java:110) >>>>>> at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273) >>>>>> at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156) >>>>>> at = org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassand= raDaemon.java:173) >>>>>> at = org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCass= andraDaemon.java:314) >>>>>> at = org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79) >>>>>> Exception encountered during startup. >>>>>>=20 >>>>>> This looks pretty much the same as issue = https://issues.apache.org/jira/browse/CASSANDRA-1889 which is in = resolution "cannot reproduce". >>>>>>=20 >>>>>> What can I do to make the node starting again? >>>>>>=20 >>>>>> Thanks in advance and best regards >>>>>> Udo >>>>>=20 >>>=20 >>=20 >=20 --Apple-Mail-4--613782371 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii I've = not heard of it happening in a production = system. 

The commit log and the schema have = diverged, that does not normally = happen. 

Aaron
 
On 29 Apr 2011, at 20:54, Subscriber wrote:

But isn't their a better way = than to delete the commitlogs?
What do I do when this happens in a = productive = system?

Ciao
Udo

= Am 29.04.2011 um 02:07 schrieb aaron morton:

Thought you may have re-created = the schema. 

Kill the process like that should = be ok, let us know if you get the error = again. 

Aaron

=
On 29 Apr 2011, at 02:56, Subscriber wrote:

Hi = Aaron, 

what exactly do you mean?
I = restarted the cluster by calling

= > bin/cassandra -p pid.file

on = all three nodes. The first node is the (only) = seed.

Udo


= Am 27.04.2011 um 23:28 schrieb Aaron Morton:

What approach did you take to restarting the = cluster? 

It looks like the keyspace name = was changed and the log replay tried to write to the old = one.

Aaron

On 28/04/2011, at 12:03 AM, = Subscriber <subscriber@zfabrik.de> = wrote:

Hi = again, 

some more remarks. 
I = renamed the commitlog directory on the third node so that cassandra = cannot see it on startup. 
Now the node starts fine. The = problem seems to have something to do with the = commitlogs...

Best = Regards
Udo


Am = 27.04.2011 um 13:22 schrieb Subscriber:

at = org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:275)
= at = org.apache.cassandra.db.Table.<init>(Table.java:221)
at = org.apache.cassandra.db.Table.open(Table.java:110)
at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:273)
= at = org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
= at = org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassand= raDaemon.java:173)
at = org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCass= andraDaemon.java:314)
at = org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)<= /div>
Exception encountered during = startup.

This looks pretty much the same = as issue https://issu= es.apache.org/jira/browse/CASSANDRA-1889 which is in resolution = "cannot reproduce".

What can I do to make the = node starting again?

Thanks in advance and best = regards
Udo





<= /div>= --Apple-Mail-4--613782371--