cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dikang Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12172) Fail to bootstrap new node.
Date Tue, 12 Jul 2016 21:55:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373779#comment-15373779
] 

Dikang Gu commented on CASSANDRA-12172:
---------------------------------------

[~jkni], thanks for looking at this:

I tried several things:
1) set larger ring_delay_ms, from 30s to 60s, which does not seem to help a lot.
2) sleep 2 mins between each bootstrap, which does not help either.
3) set the range movement to be false, which introduces a new type of error: 

{code}
2016-07-12_21:52:09.40788 INFO  21:52:09 [SharedPool-Worker-1]: InetAddress /2401:db00:2011:50c7:face:0:2d:0
is now UP
2016-07-12_21:52:09.52132 Exception (java.lang.IllegalStateException) encountered during startup:
unable to find sufficient sources for streaming range (12928845086740495435201607154872516048,12932880296782147283630181058291836395]
in keyspace system_distributed
2016-07-12_21:52:09.52496 ERROR 21:52:09 [main]: Exception encountered during startup
2016-07-12_21:52:09.52497 java.lang.IllegalStateException: unable to find sufficient sources
for streaming range (12928845086740495435201607154872516048,12932880296782147283630181058291836395]
in keyspace system_distributed
2016-07-12_21:52:09.52498       at org.apache.cassandra.dht.RangeStreamer.getRangeFetchMap(RangeStreamer.java:308)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52498       at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:155)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52498       at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52498       at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52498       at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52499       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:709)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52499       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52499       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52500       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52500       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-07-12_21:52:09.52646 WARN  21:52:09 [StorageServiceShutdownHook]: No local state or state
is in silent shutdown, not announcing shutdown
2016-07-12_21:52:09.52659 INFO  21:52:09 [StorageServiceShutdownHook]: Waiting for messaging
service to quiesce
{code}

I also sent you an email about this, let me know if you need more information.

Thanks
Dikang

> Fail to bootstrap new node.
> ---------------------------
>
>                 Key: CASSANDRA-12172
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12172
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Dikang Gu
>
> When I try to bootstrap new node in the cluster, sometimes it failed because of following
exceptions.
> {code}
> 2016-07-12_05:14:55.58509 INFO  05:14:55 [main]: JOINING: Starting to bootstrap...
> 2016-07-12_05:14:56.07491 INFO  05:14:56 [GossipTasks:1]: InetAddress /2401:db00:2011:50c7:face:0:9:0
is now DOWN
> 2016-07-12_05:14:56.32219 Exception (java.lang.RuntimeException) encountered during startup:
A node required to move the data consistently is down (/2401:db00:2011:50c7:face:0:9:0). If
you wish to move the data from a potentially inconsis
> tent replica, restart the node with -Dcassandra.consistent.rangemovement=false
> 2016-07-12_05:14:56.32582 ERROR 05:14:56 [main]: Exception encountered during startup
> 2016-07-12_05:14:56.32583 java.lang.RuntimeException: A node required to move the data
consistently is down (/2401:db00:2011:50c7:face:0:9:0). If you wish to move the data from
a potentially inconsistent replica, restart the node with -Dc
> assandra.consistent.rangemovement=false
> 2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32584       at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32584       at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32584       at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32585       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:709)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32585       at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32585       at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32586       at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32586       at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-07-12_05:14:56.32730 WARN  05:14:56 [StorageServiceShutdownHook]: No local state
or state is in silent shutdown, not announcing shutdown
> {code}
> Here are more logs: https://gist.github.com/DikangGu/c6a83eafdbc091250eade4a3bddcc40b
> I'm pretty sure there are no DOWN nodes or restarted nodes in the cluster, but I still
see a lot of nodes UP and DOWN in the gossip log, which failed the bootstrap at the end, is
this a known bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message