incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Michalski <mich...@opera.com>
Subject Re: Repair does not fix inconsistency
Date Thu, 04 Apr 2013 09:30:43 GMT
Hi Aaron,

At first, before I go with a lot of logs:

I'm considering a problem related to this issue: 
https://issues.apache.org/jira/browse/CASSANDRA-4905

Let's say the tombstone on one of the nodes (X) is gcable and was not 
compacted (purged) so far. After it was created we re-created this row, 
but due some problems it was written only to the second node (Y), so we 
have "live" data on node Y which is newer than the gcable tombstone on 
replica node X. Some time ago we did NOT repair our cluster for a  while 
(well, pretty long while), so it's possible that such situation happened.

My concern is: will AntiEntropy ignore this tombstone only, or basically 
everything related to the row key that this tombstone was created for?

If it's not the case, here are the answers you asked for :-)

> What version are you on ?

1.2.1
(plus CASSANDRA-5298 & CASSANDRA-5299 patches to be exact ;-) )

> Can you run a repair on the CF and check:
> Does the repair detect differences in the CF and stream changes ?
 > After the streaming does it run a secondary index rebuild on the new 
sstable ? (Should be in the logs)

I'm attaching a log file (cssa-repair.log).

Just to clarify: the key I use for tests belongs to *:1:7 node and *:2:1 
is a replica for that node (checked with nodetool getendpoints). 
Yesterday I was repairing this CF cluster-wide, but to (hopefully) make 
debugging simplier, what I send you is related only to these two nodes.

So as I understand these logs: no changes have been detected and nothing 
was streamed. Indexes have not been rebuilt, obviously.

However, on the other hand I'd expect to get "Nothing to repair for 
keyspace production" in nodetool output in this case - am I wrong? I'm a 
bit confused with the info I get here ;-)

> Can you provide the full query trace ?

I'm attaching two files, as this stack trace is pretty long: 
no-index.log (query by row key) and index.log (query by indexed column).


M.

Mime
View raw message