cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dikang Gu <dikan...@gmail.com>
Subject Re: Node replacement failed in 2.2
Date Mon, 21 Nov 2016 15:19:34 GMT
Hmm, I don't think we use join_ring=false or write_survey=true for that
node. I already remove_node to take the bad node out of ring, will try to
have more debug logs next time.

Thanks.

On Sun, Nov 20, 2016 at 2:31 PM, Paulo Motta <pauloricardomg@gmail.com>
wrote:

> Is there any chance the replaced node recently resumed bootstrap, joined
> with join_ring=false or write_survey=true? If so, perhaps this could be
> related to CASSANDRA-12935.
>
> Otherwise gossip tokens being empty is definitely unexpected behavior and
> you should probably file another ticket with more details/context (such as
> gossip debug logs of replacement and other nodes, and if the replacement
> node had the same or different ip as the original node since they are
> slightly different code paths after #8523).
>
> 2016-11-18 19:07 GMT-02:00 Dikang Gu <dikang85@gmail.com>:
>
>> Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows
>> "TOKENS: not present", on all live nodes. It means tokens are missing,
>> right? What would cause this?
>>
>> Thanks.
>> Dikang.
>>
>> On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pauloricardomg@gmail.com>
>> wrote:
>>
>>> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091:
>>> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is
>>> it only missing on this node or other nodes as well?
>>>
>>> 2016-11-18 17:02 GMT-02:00 Dikang Gu <dikang85@gmail.com>:
>>>
>>>> Hi, I encountered couple times that I could not replace a down node due
>>>> to error:
>>>>
>>>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
>>>> encountered during startup: Could not find tokens for
>>>> /2401:db00:2130:4091:face:0:13:0 to replace
>>>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
>>>> during startup
>>>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find
>>>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace
>>>> 2016-11-17_19:33:58.70491       at org.apache.cassandra.service.S
>>>> torageService.prepareReplacementInfo(StorageService.java:525)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492       at org.apache.cassandra.service.S
>>>> torageService.prepareToJoin(StorageService.java:760)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492       at org.apache.cassandra.service.S
>>>> torageService.initServer(StorageService.java:693)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492       at org.apache.cassandra.service.S
>>>> torageService.initServer(StorageService.java:585)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492       at org.apache.cassandra.service.C
>>>> assandraDaemon.setup(CassandraDaemon.java:300)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70493       at org.apache.cassandra.service.C
>>>> assandraDaemon.activate(CassandraDaemon.java:516)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70493       at org.apache.cassandra.service.C
>>>> assandraDaemon.main(CassandraDaemon.java:625)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70649 INFO  19:33:58 [StorageServiceShutdownHook]:
>>>> Announcing shutdown
>>>> 2016-11-17_19:34:00.70967 INFO  19:34:00 [StorageServiceShutdownHook]:
>>>> Waiting for messaging service to quiesce
>>>> 2016-11-17_19:34:00.71066 INFO  19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]:
>>>> MessagingService has terminated the accept() thread
>>>>
>>>> Did not find a relevant ticket for this, is anyone aware of this?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Dikang
>>>>
>>>>
>>>
>>
>>
>> --
>> Dikang
>>
>>
>


-- 
Dikang

Mime
View raw message