incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Michalski <mich...@opera.com>
Subject Re: Repair does not fix inconsistency
Date Thu, 04 Apr 2013 11:18:18 GMT
Hi Sylvain,

Thanks for explaination :-) However, in this case, I still do not get 
why this (probably) gcable tombstone on 2:1 could cause this mess. As AE 
ignores only the tombstone itself (which means that there are no data 
for this key on 2:1 node from repair's point of view), it should result 
in repairing this inconsistency by streaming missing data from 1:7 (thus 
there'll be both: live data and gcable tombstone on 2:1 after the 
repair), shouldn't it?

Yes, I'm considering running a major compaction (we store about 200KB of 
data in this CF, so it's not a problem at all ;-) ), but before I do I 
want to make sure I understand the problem, so as long as I can live 
with QUORUM read / writes, I'll wait with compacting and play a bit with 
this problem :-)

M.

W dniu 04.04.2013 12:28, Sylvain Lebresne pisze:
>> I'm considering a problem related to this issue:
>> https://issues.apache.org/**jira/browse/CASSANDRA-4905<https://issues.apache.org/jira/browse/CASSANDRA-4905>
>>
>> Let's say the tombstone on one of the nodes (X) is gcable and was not
>> compacted (purged) so far. After it was created we re-created this row, but
>> due some problems it was written only to the second node (Y), so we have
>> "live" data on node Y which is newer than the gcable tombstone on replica
>> node X. Some time ago we did NOT repair our cluster for a  while (well,
>> pretty long while), so it's possible that such situation happened.
>>
>
> That would my bet, yes.
>
>
> My concern is: will AntiEntropy ignore this tombstone only, or basically
>> everything related to the row key that this tombstone was created for?
>>
>
> It will only ignore the tombstone itself.
>
> In theory, that older than gcgrace tombstone should eventually be reclaimed
> by compaction, though it's not guaranteed that it will be by the first
> compaction including it (but if you use SizeTieredCompaction, a major
> compaction would ensure that you get rid of it; that being said, I'm not
> necessarily advising a major compaction, if you can afford to wait for
> normal compaction to get rid of it, that's probably simpler).
>
> --
> Sylvain
>
>
>>
>> If it's not the case, here are the answers you asked for :-)
>>
>>
>>   What version are you on ?
>>>
>>
>> 1.2.1
>> (plus CASSANDRA-5298 & CASSANDRA-5299 patches to be exact ;-) )
>>
>>
>>   Can you run a repair on the CF and check:
>>> Does the repair detect differences in the CF and stream changes ?
>>>
>>> After the streaming does it run a secondary index rebuild on the new
>> sstable ? (Should be in the logs)
>>
>> I'm attaching a log file (cssa-repair.log).
>>
>> Just to clarify: the key I use for tests belongs to *:1:7 node and *:2:1
>> is a replica for that node (checked with nodetool getendpoints). Yesterday
>> I was repairing this CF cluster-wide, but to (hopefully) make debugging
>> simplier, what I send you is related only to these two nodes.
>>
>> So as I understand these logs: no changes have been detected and nothing
>> was streamed. Indexes have not been rebuilt, obviously.
>>
>> However, on the other hand I'd expect to get "Nothing to repair for
>> keyspace production" in nodetool output in this case - am I wrong? I'm a
>> bit confused with the info I get here ;-)
>>
>>
>>   Can you provide the full query trace ?
>>>
>>
>> I'm attaching two files, as this stack trace is pretty long: no-index.log
>> (query by row key) and index.log (query by indexed column).
>>
>>
>> M.
>>
>


Mime
View raw message