incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "R. Verlangen" <ro...@us2.nl>
Subject Re: Trouble with wrong data
Date Fri, 13 Apr 2012 12:41:54 GMT
It sounds like the commitlog has been replayed however I have really no
idea whether this could have happened. Anyone?

2012/4/13 Alain RODRIGUEZ <arodrime@gmail.com>

> The commitlog_total_space_in_mb was not set, I set it to avoid having the
> same problem in the future.
>
> I am aware of the over-counting problem introduced by the counters. The
> point is that I use them to make statistics per hours. I can understand
> having some wrong counts in the column corresponding to the crash time, but
> how to explain that all my counts since the start (months ago) have become
> wrong after the crash ?
>
> After the crash I tried to repair my entire keyspace from one of the 2
> nodes and this made my server crash again, no idea why. Can this failed
> repair be at the origin of the corrupted data ?
>
> I'm still replaying all my counts of the past months and I'm afraid this
> kind of bug could happen again...
>
> I was using cassandra for months without any issue.
>
> Alain
>
> 2012/4/11 aaron morton <aaron@thelastpickle.com>
>
>> However after recovering from this issue (freeing some space and fixing
>>> the value of  "commitlog_total_space_in_mb" in cassandra.yaml)
>>>
>> Did the commit log grow larger than commitlog_total_space_in_mb ?
>>
>> I realized that all statistics were all destroyed. I have bad values on
>>> every single counter since I start using them (september) !
>>>
>> Counter operations are not idempotent. If you client retries a counter
>> operation it may result in the increment been applied twice. Could this
>> have been your issue ?
>>
>> Cheers
>>
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 11/04/2012, at 2:35 AM, Alain RODRIGUEZ wrote:
>>
>> By the way, I am using Cassandra 1.0.7, CL = ONE (R/W), RF = 2, 2 EC2
>> c1.medium nodes cluster
>>
>> Alain
>>
>> 2012/4/10 Alain RODRIGUEZ <arodrime@gmail.com>
>>
>>> Hi, I'm experimenting a strange and very annoying phenomena.
>>>
>>> I had a problem with the commit log size which grew too much and full
>>> one of the hard disks in all my nodes almost at the same time (2 nodes
>>> only, RF=2, so the 2 nodes are behaving exactly in the same way)
>>>
>>> My data are mounted in an other partition that was not full. However
>>> after recovering from this issue (freeing some space and fixing the value
>>> of  "commitlog_total_space_in_mb" in cassandra.yaml) I realized that all
>>> statistics were all destroyed. I have bad values on every single counter
>>> since I start using them (september) !
>>>
>>> Does anyone experimented something similar or have any clue on this ?
>>>
>>> Do you need more information ?
>>>
>>> Alain
>>>
>>
>>
>>
>


-- 
With kind regards,

Robin Verlangen
www.robinverlangen.nl

Mime
View raw message