incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Desimpel, Ignace" <Ignace.Desim...@nuance.com>
Subject Sporadic gossip exception on add node
Date Thu, 06 Feb 2014 11:15:29 GMT
Environment : linux, cassandra 2.0.4, 3 node, embedded, byte ordered, LCS

When I add a node to the existing 3 node cluster I sometimes get the exception 'Unable to
gossip with any seeds ' listed below. If I just restart it without any change then mostly
it works. Must be some timing issue.

The Cassandra at that time is configured using the Cassandra.yaml file
with the auto_bootstrap set true
and the initial_token set to something like : 00f35256, 041e692a, 0562d8b2, 0930274a, 0b16ce96,
0c5b3e1e, 10cac47a, 12b16bc6, 13f5db4e, 186561aa, 1907996e, 1c32b042, 1e19578e ......

The two seeds configured in this yaml are 10.164.8.250 and 10.164.8.249 and these are up and
running.
The new node to add has ip 10.164.8.93

At the time of the exception, I do not get the gossip message 'Handshaking version with /10.164.8.93'
on the seeds.
If the exception does not occurs, then I do get that gossip message 'Handshaking version with
/10.164.8.93' on the seed

2014-01-31 13:40:36.380 Loading persisted ring state
2014-01-31 13:40:36.386 Starting Messaging Service on port 9804
2014-01-31 13:40:36.408 Handshaking version with /10.164.8.250
2014-01-31 13:40:36.408 Handshaking version with /10.164.8.249
2014-01-31 13:41:07.415 Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
                at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160) ~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.startUpCassandra(CassandraStore.java:469)
[landc-services-search-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT-87937]
                at be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.run(CassandraStore.java:460)
[landc-services-search-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT-87937]
java.lang.RuntimeException: Unable to gossip with any seeds
                at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1160)
                at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:426)
                at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:618)
                at org.apache.cassandra.service.StorageService.initServer(StorageService.java:586)
                at org.apache.cassandra.service.StorageService.initServer(StorageService.java:485)
                at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
                at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
                at be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.startUpCassandra(CassandraStore.java:469)
                at be.landc.services.search.server.db.baseserver.indexsearch.store.cassandra.CassandraStore$CassThread.run(CassandraStore.java:460)
Exception encountered during startup: Unable to gossip with any seeds
2014-01-31 13:41:07.419 Exception in thread Thread[StorageServiceShutdownHook,5,main]
java.lang.NullPointerException: null
                at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:349)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.shutdownClientServers(StorageService.java:364)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService.access$3(StorageService.java:361)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:551)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.0.4-SNAPSHOT.jar:2.0.4-SNAPSHOT]
                at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40]
2014-01-31 13:41:07.420 ShutDownHook requests shutdown on be.landc.services.cdi.server.cassandra.CDIServer@7c32d1a3<mailto:be.landc.services.cdi.server.cassandra.CDIServer@7c32d1a3>
2014-01-31 13:41:07.421 Shutdown server request

Mime
View raw message