Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F069F1038B for ; Thu, 6 Feb 2014 15:49:21 +0000 (UTC) Received: (qmail 29667 invoked by uid 500); 6 Feb 2014 15:49:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 29583 invoked by uid 500); 6 Feb 2014 15:49:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 29562 invoked by uid 99); 6 Feb 2014 15:49:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 15:49:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of paulo.motta@chaordicsystems.com designates 209.85.220.43 as permitted sender) Received: from [209.85.220.43] (HELO mail-pa0-f43.google.com) (209.85.220.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 15:49:11 +0000 Received: by mail-pa0-f43.google.com with SMTP id rd3so1841517pab.30 for ; Thu, 06 Feb 2014 07:48:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=waGhFRmkLI8nInbOAzAO1y+qZqNL6GbJiEKz6Rx8bAc=; b=g5CgFMJ3J6MZLLpAjq341xczqNv1PAGNUgfwnijtRSD/hzKGvPBsYGzRiNh01ClU8E j6jxSZdBilnbR3OkjVP02CBUi0QBsjqvWh/dRhBEaexLG15Lgr6kAMQpljorIyqIcbZ/ iYKgP9jkvOKCS5IcifXWN38MRyKl2JGnTzku2AlZ55zJYiDoOxpxog/DF6HkFEY6PhkO mjcfrvtAM6/RS0lLDw3diHQ7SqQIMAMWlEsCxqzXrswxVgj2dKIlcYq4EJN/10faghe4 5mUiC27GnvN+Hjj5RgEXKWQECrd3p2AQ9X51v2UsHfAvZOppi3n6iXzgniT6xCl6cXpr GX4A== X-Gm-Message-State: ALoCoQnQnXN4woghbsE7Fg1behwYDH7kzjqHtInfUDSaH8qt4LMEM3Dm0U2DbmBCiF15Cw+EPuAW X-Received: by 10.66.146.199 with SMTP id te7mr1405618pab.106.1391701729837; Thu, 06 Feb 2014 07:48:49 -0800 (PST) MIME-Version: 1.0 Received: by 10.69.0.193 with HTTP; Thu, 6 Feb 2014 07:48:29 -0800 (PST) From: Paulo Ricardo Motta Gomes Date: Thu, 6 Feb 2014 13:48:29 -0200 Message-ID: Subject: Bootstrap failure on C* 1.2.13 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b6d99ae3d9d0104f1becff1 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d99ae3d9d0104f1becff1 Content-Type: text/plain; charset=ISO-8859-1 Hello, One the nodes of our cluster failed and I performed the "replace a dead node procedure" described in [1]. After about 5 or 6 hours of streaming during bootstrap, the node fails with the exception "Unable to fetch range for keyspace foobar from any hosts." [2]. I haven't found any thread or forum with the same error message yet. Could this be related to CASSANDRA-6648, or is this a 1.2.14 issue? * Additional info: 0) Not using vnodes. 1) The replacement node does not show up on "nodetool status" but shows up on "nodetool gossip info". 2) I am in the middle of a rebuild operation to bootstrap a new datacenter, but I'm not rebuilding any nodes on the dead node range. 3) The replacement node was using exactly the same cassandra.yaml configuration as the original node (apart from initial_token, obviously), including the same seeds' hostnames, and after talking to the nodes in the cluster, it was getting a strange "Unable to contact any seeds!" [3] message and crashing. I solved this by using actual IP addresses instead of hostnames for the seeds, but what is weird is that all nodes work well with seeds hostnames. This is probably an unrelated issue that was already solved, but just in case it's relevant. [1]: http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node [2]: INFO [Thread-8048] 2014-02-06 15:24:05,385 StreamInSession.java (line 199) Finished streaming session a3d0e841-8f11-11e3-b3d4-438819ab6fdb from / 23.23.48.71 ERROR [main] 2014-02-06 15:24:05,390 CassandraDaemon.java (line 464) Exception encountered during startup java.lang.RuntimeException: Unable to fetch range [(25136549843694323996529816280365324662,30453461826833987488145094511486702966], (42535295865117307932921825918971026432,46404197776252977962990779214850837877], (35770373809973650979760322762601081269,41087285793113314471375551001729459574], (30453461826833987488145094521416702966,31901471898837980949691369441728269824], (41087285793113314471375551003729159574,42535295865117307932921825128971026432]] for keyspace foobar from any hosts at org.apache.cassandra.dht.RangeStreamer.fetch(RangeStreamer.java:260) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:84) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490) ERROR [StorageServiceShutdownHook] 2014-02-06 15:24:05,404 CassandraDaemon.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358) at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373) at org.apache.cassandra.service.StorageService.access$000(StorageService.java:89) at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:662) [3]: ERROR [main] 2014-02-06 01:04:35,715 CassandraDaemon.java (line 464) Exception encountered during startup java.lang.IllegalStateException: Unable to contact any seeds! at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:977) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:483) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348) -- *Paulo Motta* Chaordic | *Platform* *www.chaordic.com.br * +55 48 3232.3200 +55 83 9690-1314 --047d7b6d99ae3d9d0104f1becff1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hello,

One the nodes of our = cluster failed and I performed the "replace a dead node procedure"= ; described in [1]. After about 5 or 6 hours of streaming during bootstrap,= the node fails with the exception "Unable to fetch range for keyspace= foobar from any hosts." [2].

I haven't found any thread or forum with the same error messag= e yet. Could this be related to CASSANDRA-6648, or is this a 1.2.14 issue?<= /div>

* Additional info:

0) Not using vnod= es.

1) The replacement node does not show up on "nodet= ool status" but shows up on "nodetool gossip info".

2) I am in the middle of a rebuild operation to bootstrap = a new datacenter, but I'm not rebuilding any nodes on the dead node ran= ge.

3) The replacement node was using exactly the same cass= andra.yaml configuration as the original node (apart from initial_token, ob= viously), including the same seeds' hostnames, and after talking to the= nodes in the cluster, it was getting a strange "Unable to contact any= seeds!" [3] message and crashing. I solved this by using actual IP ad= dresses instead of hostnames for the seeds, but what is weird is that all n= odes work well with seeds hostnames. This is probably an unrelated issue th= at was already solved, but just in case it's relevant.=A0


[2]= :=A0

=A0INFO [Thread-8048] 2014-02-06 15:24:05,385 Stre= amInSession.java (line 199) Finished streaming session a3d0e841-8f11-11e3-b= 3d4-438819ab6fdb from /23.23.48.71
ERROR [main] 2014-02-06 15:24:05,390 CassandraDaemon.java (line 464) E= xception encountered during startup
java.lang.RuntimeException: U= nable to fetch range [(25136549843694323996529816280365324662,3045346182683= 3987488145094511486702966], (42535295865117307932921825918971026432,4640419= 7776252977962990779214850837877], (35770373809973650979760322762601081269,4= 1087285793113314471375551001729459574], (3045346182683398748814509452141670= 2966,31901471898837980949691369441728269824], (4108728579311331447137555100= 3729159574,42535295865117307932921825128971026432]] for keyspace foobar fro= m any hosts
=A0 =A0 =A0 =A0 at org.apache.cassandra.dht.RangeStreamer.fetch(RangeS= treamer.java:260)
=A0 =A0 =A0 =A0 at org.apache.cassandra.dht.Boo= tStrapper.bootstrap(BootStrapper.java:84)
=A0 =A0 =A0 =A0 at org.= apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)<= /div>
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.StorageService.joinTok= enRing(StorageService.java:745)
=A0 =A0 =A0 =A0 at org.apache.cas= sandra.service.StorageService.initServer(StorageService.java:586)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.StorageService.initServer(= StorageService.java:483)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.CassandraDaemon.setup(= CassandraDaemon.java:348)
=A0 =A0 =A0 =A0 at org.apache.cassandra= .service.CassandraDaemon.activate(CassandraDaemon.java:447)
=A0 = =A0 =A0 =A0 at org.apache.cassandra.service.CassandraDaemon.main(CassandraD= aemon.java:490)
ERROR [StorageServiceShutdownHook] 2014-02-06 15:24:05,404 CassandraDa= emon.java (line 191) Exception in thread Thread[StorageServiceShutdownHook,= 5,main]
java.lang.NullPointerException
=A0 =A0 =A0 =A0 = at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageS= ervice.java:358)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.StorageService.shutdow= nClientServers(StorageService.java:373)
=A0 =A0 =A0 =A0 at org.ap= ache.cassandra.service.StorageService.access$000(StorageService.java:89)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.StorageService$1.ru= nMayThrow(StorageService.java:551)
=A0 =A0 =A0 =A0 at org.apache.cassandra.utils.WrappedRunnable.run(Wrap= pedRunnable.java:28)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(Thre= ad.java:662)


[3]:=A0

ERROR [main] 2014-02-06 01:04:35,715 CassandraDaemon.java (line 464) E= xception encountered during startup
java.lang.IllegalStateExcepti= on: Unable to contact any seeds!
at org.apache.cassandra.service.StorageService.bootstrap= (StorageService.java:977)
at org.apache.cassa= ndra.service.StorageService.joinTokenRing(StorageService.java:745)
at org.apache.cassandr= a.service.StorageService.initServer(StorageService.java:586)
at org.apache.cassa= ndra.service.StorageService.initServer(StorageService.java:483)
<= span class=3D"" style=3D"white-space:pre"> at org.apache.cassandra.s= ervice.CassandraDaemon.setup(CassandraDaemon.java:348)


--
Paulo Motta

Chaordic | Platform
www.chaordic.com.br
+55 48 3232.3200
+55 83 9690-1314
--047d7b6d99ae3d9d0104f1becff1--