cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Node forgets about most of its column families
Date Thu, 30 Aug 2012 05:26:27 GMT
For those playing along at home Edwards ticket was marked as a dup of

Problem with creating keyspace after drop
https://issues.apache.org/jira/browse/CASSANDRA-4219

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 30/08/2012, at 4:43 AM, Edward Sargisson <edward.sargisson@globalrelay.net> wrote:

> Hi Aaron,
> Thanks for the reply. I've recorded what we know at https://issues.apache.org/jira/browse/CASSANDRA-4583.
> This includes log snippets from two of the nodes from around the time. I don't know what
is relevant so they've got everything that was in the system log at the time of the failure
and recovery.
> 
> Nodetool crashed but not returning, having nothing appear in the logs and nodetool compactionstats
and nodetool netstats indicating that nothing was happening.
> 
> Thanks for your time looking at this.
> 
> Cheers,
> Edward
> 
> 
> On 12-08-29 02:44 AM, aaron morton wrote:
>>> But the following nodetool repair crashes. It has to be stopped and then re-started.
>> How did it crash ?
>> 
>>> Are there any suggestions for logging or similar so that we can get a clue next
time this happens.
>> Can you make the logs from #5 available?
>> 
>> If you feel you can describe the situation please create a ticket on https://issues.apache.org/jira/browse/CASSANDRA
>> 
>> Cheers
>> 
>>  
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 29/08/2012, at 8:38 AM, Edward Sargisson <edward.sargisson@globalrelay.net>
wrote:
>> 
>>> For the record, we just had a recurrence of this. 
>>> This time, when the node (#5) came back it didn't properly rejoin the ring. 
>>> We stopped every node and brought them back one by one to get the ring to link
up correctly.
>>> Then, all the even nodes (#2, #4, #6) had out of data schemas.
>>> 
>>> nodetool resetlocalschema works.
>>> But the following nodetool repair crashes. It has to be stopped and then re-started.
>>> 
>>> Are there any suggestions for logging or similar so that we can get a clue next
time this happens.
>>> 
>>> Cheers,
>>> Edward
>>> 
>>> 
>>> On 12-08-24 11:18 AM, Edward Sargisson wrote:
>>>> Sadly, I don't think we can get much.
>>>> 
>>>> All I know about the repro is that it was around a node restart. I've just
tried that and everything's fine. I see now ERROR level messages in the logs.
>>>> 
>>>> Clearly, some other conditions are required but we don't know them as yet.
>>>> 
>>>> Many thanks,
>>>> Edward
>>>> 
>>>> 
>>>> On 12-08-24 03:29 AM, aaron morton wrote:
>>>>> If this is still a test environment can you try to reproduce the fault
? Or provide some more details on the sequence of events?
>>>>> 
>>>>> If you still have the logs around can you see if any ERROR level messages
were logged?
>>>>> 
>>>>> Cheers
>>>>> 
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>> 
>>>>> On 24/08/2012, at 8:33 AM, Edward Sargisson <edward.sargisson@globalrelay.net>
wrote:
>>>>> 
>>>>>> Ah, yes, I forgot that bit thanks!
>>>>>> 
>>>>>> 1.1.2 running on Centos.
>>>>>> 
>>>>>> Running nodetool resetlocalschema then nodetool repair fixed the
problem but not understanding what happened is a concern.
>>>>>> 
>>>>>> Cheers,
>>>>>> Edward
>>>>>> 
>>>>>> 
>>>>>> On 12-08-23 12:40 PM, Rob Coli wrote:
>>>>>>> On Thu, Aug 23, 2012 at 11:47 AM, Edward Sargisson
>>>>>>> <edward.sargisson@globalrelay.net> wrote:
>>>>>>>> I was wondering if anybody had seen the following behaviour
before and how
>>>>>>>> we might detect it and keep the application running.
>>>>>>> I don't know the answer to your problem, but anyone who does
will want
>>>>>>> to know in what version of Cassandra you are encountering this
issue.
>>>>>>> :)
>>>>>>> 
>>>>>>> =Rob
>>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> Edward Sargisson
>>>>>> senior java developer
>>>>>> Global Relay
>>>>>> 
>>>>>> edward.sargisson@globalrelay.net
>>>>>> 
>>>>>> 
>>>>>> 866.484.6630 
>>>>>> New York | Chicago | Vancouver  |  London  (+44.0800.032.9829)  |
 Singapore  (+65.3158.1301)
>>>>>> 
>>>>>> Global Relay Archive supports email, instant messaging, BlackBerry,
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more. 
>>>>>> 
>>>>>> Ask about Global Relay Message — The Future of Collaboration in
the Financial Services World
>>>>>> 
>>>>>> All email sent to or from this address will be retained by Global
Relay’s email archiving system. This message is intended only for the use of the individual
or entity to which it is addressed, and may contain information that is privileged, confidential,
and exempt from disclosure under applicable law.  Global Relay will not be liable for any
compliance or technical information provided herein.  All trademarks are the property of their
respective owners.
>>>>> 
>>>> 
>>>> -- 
>>>> Edward Sargisson
>>>> senior java developer
>>>> Global Relay
>>>> 
>>>> edward.sargisson@globalrelay.net
>>>> 
>>>> 
>>>> 866.484.6630 
>>>> New York | Chicago | Vancouver  |  London  (+44.0800.032.9829)  |  Singapore
 (+65.3158.1301)
>>>> 
>>>> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg,
Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and                    
      more. 
>>>> 
>>>> Ask about Global Relay Message — The Future of Collaboration in the Financial
Services World
>>>> 
>>>> All email sent to or from this address will be retained by Global Relay’s
email archiving system. This message is intended only for the use of the individual or entity
to which it is addressed, and may contain information that is privileged, confidential, and
exempt from disclosure under applicable law.  Global Relay will not be liable for any compliance
or technical information provided herein.  All trademarks are the property of their respective
owners.
>>> 
>>> -- 
>>> Edward Sargisson
>>> senior java developer
>>> Global Relay
>>> 
>>> edward.sargisson@globalrelay.net
>>> 
>>> 
>>> 866.484.6630 
>>> New York | Chicago | Vancouver  |  London  (+44.0800.032.9829)  |  Singapore
 (+65.3158.1301)
>>> 
>>> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg,
Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more. 
>>> 
>>> Ask about Global Relay Message — The Future of Collaboration in the Financial
Services World
>>> 
>>> All email sent to or from this address will be retained by Global Relay’s email
archiving system. This message is intended only for the use of the individual or entity to
which it is addressed, and may contain information that is privileged, confidential, and exempt
from disclosure under applicable law.  Global Relay will not be liable for any compliance
or technical information provided herein.  All trademarks are the property of their respective
owners.
>> 
> 
> -- 
> Edward Sargisson
> senior java developer
> Global Relay
> 
> edward.sargisson@globalrelay.net
> 
> 
> 866.484.6630 
> New York | Chicago | Vancouver  |  London  (+44.0800.032.9829)  |  Singapore  (+65.3158.1301)
> 
> Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson
Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more. 
> 
> Ask about Global Relay Message — The Future of Collaboration in the Financial Services
World
> 
> All email sent to or from this address will be retained by Global Relay’s email archiving
system. This message is intended only for the use of the individual or entity to which it
is addressed, and may contain information that is privileged, confidential, and exempt from
disclosure under applicable law.  Global Relay will not be liable for any compliance or technical
information provided herein.  All trademarks are the property of their respective owners.


Mime
View raw message