cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Mallassi <omalla...@octo.com>
Subject Re: Unreachable node, not in nodetool ring
Date Thu, 19 Jul 2012 08:45:21 GMT
I got that a couple of time (due to DNS issues in our infra)

what you could try
- check in cassandra log. It is possible you see a log line telling
you 10.56.62.211
and 10.59.21.241 o 10.58.83.109  share the same token
- if 10.56.62.211 is up, try decommission (via nodetool)
- if not, move 10.59.21.241 or 10.58.83.109 to current token + 1
- use removetoken (via nodetool) to remove the token associated with
10.56.62.211.
in case of failure, you can use removetoken -f instead.

then, the unreachable IP should have disappeared.


HTH

On Thu, Jul 19, 2012 at 10:38 AM, Alain RODRIGUEZ <arodrime@gmail.com>wrote:

> Hi,
>
> I tried to add a node a few days ago and it failed. I finally made it
> work with an other node but now when I describe cluster on cli I got
> this :
>
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.Ec2Snitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions:
>       UNREACHABLE: [10.56.62.211]
>       e7e0ec6c-616e-32e7-ae29-40eae2b82ca8: [10.59.21.241, 10.58.83.109]
>
> And nodetool ring gives me :
>
> Address         DC          Rack        Status State   Load
> Owns                Token
>
>                     85070591730234615865843651857942052864
> 10.59.21.241    eu-west     1b          Up     Normal  101.17 GB
> 50.00%              0
> 10.58.83.109    eu-west     1b          Up     Normal  55.27 GB
> 50.00%              85070591730234615865843651857942052864
>
> The point, as you can see, is that one of my node has twice the
> information of the second one. I have a RF = 2 defined.
>
> My guess is that the token 0 node keep data for the unreachable node.
>
> The IP of the unreachable node doesn't belong to me anymore, I have no
> access to this ghost node.
>
> Does someone know how to completely remove this ghost node from my cluster
> ?
>
> Thank you.
>
> Alain
>
> INFO :
>
> On ubuntu (AMI Datastax 2.1 and 2.2)
> Cassandra 1.1.2 (upgraded from 1.0.9)
> 2 node cluster (+ the ghost one)
> RF = 2
>



-- 
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris

Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01

http://www.octo.com
Octo Talks! http://blog.octo.com

Mime
View raw message