cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Tkachenko <ro...@mailgunhq.com>
Subject Re: Deleted columns reappear after "repair"
Date Tue, 24 Mar 2015 18:00:39 GMT
Well, as I mentioned in my original email all machines running Cassandra
are running NTP. This was one of the first things I verified and I triple
checked that they all show the same time. Is this sufficient to ensure
clocks are synched between the nodes?

I have increased gc_grace to 100 days for now and am running repair on the
affected keyspace, it should be done today. In the meanwhile if you (or
anyone else) have other ideas / suggestions on how to debug this, they're
much appreciated.

Thanks for your help!

Roman

On Tue, Mar 24, 2015 at 10:39 AM, Duncan Sands <duncan.sands@gmail.com>
wrote:

> Hi Roman,
>
> On 24/03/15 18:05, Roman Tkachenko wrote:
>
>> Hi Duncan,
>>
>> Thanks for the response!
>>
>> I can try increasing gc_grace_seconds and run repair on all nodes. It
>> does not
>> make sense though why all *new* deletes (for the same column that
>> resurrects
>> after repair) I do are forgotten as well after repair? Doesn't Cassandra
>> insert
>> a new tombstone every time delete happens?
>>
>
> it does.  Maybe the data you are trying to delete has a timestamp
> (writetime) in the future, for example because clocks aren't synchronized
> between your nodes.
>
>
>> Also, how do I find out the value to set gc_grace_seconds to?
>>
>
> It needs to be big enough that you are sure to repair your entire cluster
> in less than that time.  For example, observe how long repairing the entire
> cluster takes and multiply by 3 or 4 (in case a repair fails or is
> interrupted one day).
>
> Once incremental repair is solid maybe the whole gc_grace thing will
> eventually go away, eg by modifying C* to only drop known repaired
> tombstones.
>
> Ciao, Duncan.
>
>
>> Thanks.
>>
>> On Tue, Mar 24, 2015 at 9:38 AM, Duncan Sands <duncan.sands@gmail.com
>> <mailto:duncan.sands@gmail.com>> wrote:
>>
>>     Hi Roman,
>>
>>     On 24/03/15 17:32, Roman Tkachenko wrote:
>>
>>         Hey guys,
>>
>>         Has anyone seen anything like this behavior or has an explanation
>> for it? If
>>         not, I think I'm gonna file a bug report.
>>
>>
>>     this can happen if repair is run after the tombstone gc_grace_period
>> has
>>     expired.  I suggest you increase gc_grace_period.
>>
>>     Ciao, Duncan.
>>
>>
>>
>

Mime
View raw message