cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Re : decommissioned nodes shows up in "nodetool describecluster" as UNREACHABLE in 2.1.12 version
Date Wed, 17 Feb 2016 10:42:07 GMT
Hi,

nodetool gossipinfo shows the decommissioned nodes as "LEFT"


I believe this is the expected behavior, we keep some a trace of leaving
nodes for a few days, this shouldn't be an issue for you

nodetool describecluster shows the decommissioned nodes as UNREACHABLE.
>

This is a weird behaviour I haven't see for a while. You might want to dig
this some more.

Restarting the entire cluster,  everytime a node is decommissioned does not
> seem right
>

Meanwhile, if you are sure the node is out and streams have ended, I guess
it could be ok to use a JMX client (MX4J, JConsole...) and then use the JMX
method Gossiper.unsafeAssassinateEndpoints(ip_address) to assassinate the
gone node from any of the remaining nodes.

How to -->
http://tumblr.doki-pen.org/post/22654515359/assassinating-cassandra-nodes
(3 years old post, I partially read it, but I think it might still be
relevant)

Has anybody experienced similar behaviour


FTR, 3 years old similar issue I faced -->
http://grokbase.com/t/cassandra/user/127knx7nn0/unreachable-node-not-in-nodetool-ring

FWIW, people using C* = 3.x, this is exposed through nodetool -->
https://docs.datastax.com/en/cassandra/3.x/cassandra/tools/toolsAssassinate.html

Keep in mind that something called 'unsafe' and 'assassinate' at the same
time is not something you want to use in a regular decommissioning process
as it drop the node with no file transfer, you basically totally lose a
node (unless node is out already which seems to be your case, it should be
safe to use it in your case). I only used it to fix gossip status in the
past or at some point when forcing a removenode was not working, followed
by full repairs on remaining nodes.

C*heers,
-----------------
Alain Rodriguez
France

The Last Pickle
http://www.thelastpickle.com

2016-02-16 20:08 GMT+01:00 sai krishnam raju potturi <pskraju88@gmail.com>:

> hi;
>     we have a 12 node cluster across 2 datacenters. We are currently using
> cassandra 2.1.12 version.
>
> SNITCH : GossipingPropertyFileSnitch
>
> When we decommissioned few nodes in a particular datacenter and observed
> the following :
>
> nodetool status shows only the live nodes in the cluster.
>
> nodetool describecluster shows the decommissioned nodes as UNREACHABLE.
>
> nodetool gossipinfo shows the decommissioned nodes as "LEFT"
>
>
> When the live nodes were restarted, "nodetool describecluster" shows only
> the live nodes, which is expected.
>
> Purging the gossip info too did not help.
>
> INFO  17:27:07 InetAddress /X.X.X.X is now DOWN
> INFO  17:27:07 Removing tokens [125897680671740685543105407593050165202,
> 140213388002871593911508364312533329916,
>  98576967436431350637134234839492449485] for /X.X.X.X
> INFO  17:27:07 InetAddress /X.X.X.X is now DOWN
> INFO  17:27:07 Removing tokens [11116977666116265389022494863106850615,
> 111270759969411259938117902792984586225,
> 138611464975439236357814418845450428175] for /X.X.X.X
>
> Has anybody experienced similar behaviour. Restarting the entire cluster,
>  everytime a node is decommissioned does not seem right. Thanks in advance
> for the help.
>
>
> thanks
> Sai
>
>
>

Mime
View raw message