incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Ricardo Motta Gomes <paulo.mo...@chaordicsystems.com>
Subject Bootstrap failure on C* 1.2.13
Date Thu, 06 Feb 2014 15:48:29 GMT
Hello,

One the nodes of our cluster failed and I performed the "replace a dead
node procedure" described in [1]. After about 5 or 6 hours of streaming
during bootstrap, the node fails with the exception "Unable to fetch range
for keyspace foobar from any hosts." [2].

I haven't found any thread or forum with the same error message yet. Could
this be related to CASSANDRA-6648, or is this a 1.2.14 issue?

* Additional info:

0) Not using vnodes.

1) The replacement node does not show up on "nodetool status" but shows up
on "nodetool gossip info".

2) I am in the middle of a rebuild operation to bootstrap a new datacenter,
but I'm not rebuilding any nodes on the dead node range.

3) The replacement node was using exactly the same cassandra.yaml
configuration as the original node (apart from initial_token, obviously),
including the same seeds' hostnames, and after talking to the nodes in the
cluster, it was getting a strange "Unable to contact any seeds!" [3]
message and crashing. I solved this by using actual IP addresses instead of
hostnames for the seeds, but what is weird is that all nodes work well with
seeds hostnames. This is probably an unrelated issue that was already
solved, but just in case it's relevant.

[1]:
http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node

[2]:

 INFO [Thread-8048] 2014-02-06 15:24:05,385 StreamInSession.java (line 199)
Finished streaming session a3d0e841-8f11-11e3-b3d4-438819ab6fdb from /
23.23.48.71
ERROR [main] 2014-02-06 15:24:05,390 CassandraDaemon.java (line 464)
Exception encountered during startup
java.lang.RuntimeException: Unable to fetch range
[(25136549843694323996529816280365324662,30453461826833987488145094511486702966],
(42535295865117307932921825918971026432,46404197776252977962990779214850837877],
(35770373809973650979760322762601081269,41087285793113314471375551001729459574],
(30453461826833987488145094521416702966,31901471898837980949691369441728269824],
(41087285793113314471375551003729159574,42535295865117307932921825128971026432]]
for keyspace foobar from any hosts
        at
org.apache.cassandra.dht.RangeStreamer.fetch(RangeStreamer.java:260)
        at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:84)
        at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:979)
        at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
        at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
        at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
        at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)
        at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:447)
        at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:490)
ERROR [StorageServiceShutdownHook] 2014-02-06 15:24:05,404
CassandraDaemon.java (line 191) Exception in thread
Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException
        at
org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:358)
        at
org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:373)
        at
org.apache.cassandra.service.StorageService.access$000(StorageService.java:89)
        at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.lang.Thread.run(Thread.java:662)


[3]:

ERROR [main] 2014-02-06 01:04:35,715 CassandraDaemon.java (line 464)
Exception encountered during startup
java.lang.IllegalStateException: Unable to contact any seeds!
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:977)
 at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:745)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
 at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:483)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:348)


-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
+55 83 9690-1314

Mime
View raw message