cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "R. Verlangen" <ro...@us2.nl>
Subject Re: Node joining / unknown
Date Thu, 08 Mar 2012 18:03:46 GMT
It seemed that one of the other nodes had trouble with a compaction task.
The C node was waiting for that.

It's now streaming all it's data into place.

Thank you all for your time!

2012/3/7 <igor@4friends.od.ua>

> just run "nodetool compactionstat" on other nodes.
>
>
> -----Original Message-----
> From: "R. Verlangen" <robin@us2.nl>
> To: user@cassandra.apache.org
> Sent: Wed, 07 Mar 2012 23:09
> Subject: Re: Node joining / unknown
>
> @Brandon: Thank you for the information. I'll do that next time.
>
> @Igor: Any ways to find out whether that is the current state? And if so,
> how to solve it?
>
> 2012/3/7 <igor@4friends.od.ua>
>
>> Maybe it wait for verification compaction on other node?
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: "R. Verlangen" <robin@us2.nl>
>> To: user@cassandra.apache.org
>> Sent: Wed, 07 Mar 2012 22:15
>> Subject: Re: Node joining / unknown
>>
>> At this moment the node has joined the ring (after a restart: tried that
>> before, but now it had finally result).
>>
>> When I try to run repair on the new node, the log says (the new node is
>> NODE C):
>>
>> INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,453 AntiEntropyService.java
>> (line 190) [repair #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle
>> tree for StorageMeta from NODE A
>>  INFO [AntiEntropyStage:1] 2012-03-07 21:12:06,643
>> AntiEntropyService.java (line 190) [repair
>> #cfcc12b0-6891-11e1-0000-70a329caccff] Received merkle tree for StorageMeta
>> from NODE B
>>
>> And then doesn't do anything anymore. Tried it a couple of times again.
>> It's just not starting.
>>
>> Results from netstats on NODE C:
>>
>> Mode: NORMAL
>> Not sending any streams.
>> Not receiving any streams.
>> Pool Name                    Active   Pending      Completed
>> Commands                        n/a         0              5
>> Responses                       n/a        93           4296
>>
>>
>> Any suggestions?
>>
>> Thank you!
>>
>> 2012/3/7 aaron morton <aaron@thelastpickle.com>
>>
>>> - When I try to remove the token, it says: Exception in thread "main"
>>> java.lang.UnsupportedOperationException: Token not found.
>>>
>>> Am assuming you ran nodetool removetoken on a node other than the
>>> joining node? What  did nodetool ring look like on that machine ?
>>>
>>> Take a look at nodetool netstats on the joining node to see if streaming
>>> has failed. If it's dead then…
>>>
>>> 1) Try restarting the joining node and run nodetool repair on it
>>> immediately. Note: am assuming QUOURM CL otherwise things may get
>>> inconsistent.
>>> or
>>> 2) Stop the node. Try to get remove the token again from another node.
>>> Node that removing a token will stream data around the place as well.
>>>
>>> Cheers
>>>
>>>   -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 7/03/2012, at 9:11 PM, R. Verlangen wrote:
>>>
>>> Hi there,
>>>
>>> I'm currently in a really weird situation.
>>> - Nodetool ring says node X is joining (this already takes 12 hours,
>>> with no activity)
>>> - When I try to remove the token, it says: Exception in thread "main"
>>> java.lang.UnsupportedOperationException: Token not found.
>>> - Removetoken status = No token removals in process.
>>>
>>> How to get that node out of my cluster?
>>>
>>> With kind regards,
>>> Robin Verlangen
>>>
>>>
>>>
>>
>

Mime
View raw message