cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Sargisson <edward.sargis...@globalrelay.net>
Subject Re: Node forgets about most of its column families
Date Tue, 28 Aug 2012 20:38:54 GMT
For the record, we just had a recurrence of this.
This time, when the node (#5) came back it didn't properly rejoin the ring.
We stopped every node and brought them back one by one to get the ring 
to link up correctly.
Then, all the even nodes (#2, #4, #6) had out of data schemas.

nodetool resetlocalschema works.
But the following nodetool repair crashes. It has to be stopped and then 
re-started.

Are there any suggestions for logging or similar so that we can get a 
clue next time this happens.

Cheers,
Edward


On 12-08-24 11:18 AM, Edward Sargisson wrote:
> Sadly, I don't think we can get much.
>
> All I know about the repro is that it was around a node restart. I've 
> just tried that and everything's fine. I see now ERROR level messages 
> in the logs.
>
> Clearly, some other conditions are required but we don't know them as yet.
>
> Many thanks,
> Edward
>
>
> On 12-08-24 03:29 AM, aaron morton wrote:
>> If this is still a test environment can you try to reproduce the 
>> fault ? Or provide some more details on the sequence of events?
>>
>> If you still have the logs around can you see if any ERROR level 
>> messages were logged?
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 24/08/2012, at 8:33 AM, Edward Sargisson 
>> <edward.sargisson@globalrelay.net 
>> <mailto:edward.sargisson@globalrelay.net>> wrote:
>>
>>> Ah, yes, I forgot that bit thanks!
>>>
>>> 1.1.2 running on Centos.
>>>
>>> Running nodetool resetlocalschema then nodetool repair fixed the 
>>> problem but not understanding what happened is a concern.
>>>
>>> Cheers,
>>> Edward
>>>
>>>
>>> On 12-08-23 12:40 PM, Rob Coli wrote:
>>>> On Thu, Aug 23, 2012 at 11:47 AM, Edward Sargisson
>>>> <edward.sargisson@globalrelay.net>  wrote:
>>>>> I was wondering if anybody had seen the following behaviour before and
how
>>>>> we might detect it and keep the application running.
>>>> I don't know the answer to your problem, but anyone who does will want
>>>> to know in what version of Cassandra you are encountering this issue.
>>>> :)
>>>>
>>>> =Rob
>>>>
>>>
>>> -- 
>>>
>>> Edward Sargisson
>>>
>>> senior java developer
>>> Global Relay
>>>
>>> edward.sargisson@globalrelay.net 
>>> <mailto:edward.sargisson@globalrelay.net>
>>>
>>>
>>> *866.484.6630*
>>> New York | Chicago | Vancouver | London (+44.0800.032.9829) | 
>>> Singapore (+65.3158.1301)
>>>
>>> Global Relay Archive supports email, instant messaging, BlackBerry, 
>>> Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
>>> Facebook and more.
>>>
>>>
>>> Ask about *Global Relay Message* 
>>> <http://www.globalrelay.com/services/message>*— *The Future of 
>>> Collaboration in the Financial Services World
>>>
>>> *
>>> *All email sent to or from this address will be retained by Global 
>>> Relay’s email archiving system. This message is intended only for 
>>> the use of the individual or entity to which it is addressed, and 
>>> may contain information that is privileged, confidential, and exempt 
>>> from disclosure under applicable law.  Global Relay will not be 
>>> liable for any compliance or technical information provided herein.  
>>> All trademarks are the property of their respective owners.
>>>
>>
>
> -- 
>
> Edward Sargisson
>
> senior java developer
> Global Relay
>
> edward.sargisson@globalrelay.net <mailto:edward.sargisson@globalrelay.net>
>
>
> *866.484.6630*
> New York | Chicago | Vancouver | London (+44.0800.032.9829) | 
> Singapore (+65.3158.1301)
>
> Global Relay Archive supports email, instant messaging, BlackBerry, 
> Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
> Facebook and more.
>
>
> Ask about *Global Relay Message* 
> <http://www.globalrelay.com/services/message>*— *The Future of 
> Collaboration in the Financial Services World
>
> *
> *All email sent to or from this address will be retained by Global 
> Relay’s email archiving system. This message is intended only for the 
> use of the individual or entity to which it is addressed, and may 
> contain information that is privileged, confidential, and exempt from 
> disclosure under applicable law.  Global Relay will not be liable for 
> any compliance or technical information provided herein.  All 
> trademarks are the property of their respective owners.
>

-- 

Edward Sargisson

senior java developer
Global Relay

edward.sargisson@globalrelay.net <mailto:edward.sargisson@globalrelay.net>


*866.484.6630*
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore 
(+65.3158.1301)

Global Relay Archive supports email, instant messaging, BlackBerry, 
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
Facebook and more.


Ask about *Global Relay Message* 
<http://www.globalrelay.com/services/message>*— *The Future of 
Collaboration in the Financial Services World

*
*All email sent to or from this address will be retained by Global 
Relay’s email archiving system. This message is intended only for the 
use of the individual or entity to which it is addressed, and may 
contain information that is privileged, confidential, and exempt from 
disclosure under applicable law.  Global Relay will not be liable for 
any compliance or technical information provided herein. All trademarks 
are the property of their respective owners.


Mime
View raw message