incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Unreachable node, not in nodetool ring
Date Thu, 19 Jul 2012 09:13:40 GMT
Hi, I wasn't able to see the token used currently by the 10.56.62.211
(ghost node).

I already removed the token 6 days ago :

-> "Removing token 170141183460469231731687303715884105727 for /10.56.62.211"

"- check in cassandra log. It is possible you see a log line telling
you 10.56.62.211 and 10.59.21.241 o 10.58.83.109  share the same
token"

Nothing like that in the logs

I tried the following without success :

$ nodetool -h localhost removetoken 170141183460469231731687303715884105727
Exception in thread "main" java.lang.UnsupportedOperationException:
Token not found.
...

I really thought this was going to work :-).

Any other ideas ?

Alain

PS : I heard that Octo is a nice company and you use Cassandra so I
guess you're fine in there :-). I wish you the best thanks for your
help.

2012/7/19 Olivier Mallassi <omallassi@octo.com>:
> I got that a couple of time (due to DNS issues in our infra)
>
> what you could try
> - check in cassandra log. It is possible you see a log line telling you
> 10.56.62.211 and 10.59.21.241 o 10.58.83.109  share the same token
> - if 10.56.62.211 is up, try decommission (via nodetool)
> - if not, move 10.59.21.241 or 10.58.83.109 to current token + 1
> - use removetoken (via nodetool) to remove the token associated with
> 10.56.62.211. in case of failure, you can use removetoken -f instead.
>
> then, the unreachable IP should have disappeared.
>
>
> HTH
>
> On Thu, Jul 19, 2012 at 10:38 AM, Alain RODRIGUEZ <arodrime@gmail.com>
> wrote:
>>
>> Hi,
>>
>> I tried to add a node a few days ago and it failed. I finally made it
>> work with an other node but now when I describe cluster on cli I got
>> this :
>>
>> Cluster Information:
>>    Snitch: org.apache.cassandra.locator.Ec2Snitch
>>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>>    Schema versions:
>>       UNREACHABLE: [10.56.62.211]
>>       e7e0ec6c-616e-32e7-ae29-40eae2b82ca8: [10.59.21.241, 10.58.83.109]
>>
>> And nodetool ring gives me :
>>
>> Address         DC          Rack        Status State   Load
>> Owns                Token
>>
>>                     85070591730234615865843651857942052864
>> 10.59.21.241    eu-west     1b          Up     Normal  101.17 GB
>> 50.00%              0
>> 10.58.83.109    eu-west     1b          Up     Normal  55.27 GB
>> 50.00%              85070591730234615865843651857942052864
>>
>> The point, as you can see, is that one of my node has twice the
>> information of the second one. I have a RF = 2 defined.
>>
>> My guess is that the token 0 node keep data for the unreachable node.
>>
>> The IP of the unreachable node doesn't belong to me anymore, I have no
>> access to this ghost node.
>>
>> Does someone know how to completely remove this ghost node from my cluster
>> ?
>>
>> Thank you.
>>
>> Alain
>>
>> INFO :
>>
>> On ubuntu (AMI Datastax 2.1 and 2.2)
>> Cassandra 1.1.2 (upgraded from 1.0.9)
>> 2 node cluster (+ the ghost one)
>> RF = 2
>
>
>
>
> --
> ............................................................
> Olivier Mallassi
> OCTO Technology
> ............................................................
> 50, Avenue des Champs-Elysées
> 75008 Paris
>
> Mobile: (33) 6 28 70 26 61
> Tél: (33) 1 58 56 10 00
> Fax: (33) 1 58 56 10 01
>
> http://www.octo.com
> Octo Talks! http://blog.octo.com
>
>

Mime
View raw message