First I need to vent.
One of my cassandra cluster is a dual data center setup, with DC1 acting as primary, and DC2 acting as a hot backup.
Well, guess what ? I am pretty sure that it falls behind on replication. So I am told I need to run repair.
I run repair (with -pr) on DC2. First time I run it it gets *stuck* (i.e. frozen) within the first 30 seconds, with no error or any sort of message. I then run it again -- and it completes in seconds on each node, with about 50 gigs of data on each.
That seems suspicious, so I do some research.
I am told on IRC that running repair -pr will only do the repair on "100" tokens (the offset from DC1 to DC2)… Seriously ???
Repair process is, indeed, a joke: https://issues.apache.org/jira/browse/CASSANDRA-5396 . Repair is the worst thing you can do to your cluster, it consumes enormous resources, and can leave your cluster in an inconsistent state. Oh and by the way you must run it every week…. Whoever invented that process must not live in a real world, with real applications.
No… lets have a constructive conversation.
How do I know, with certainty, that my DC2 cluster is up to date on replication ? I have a few options:
1) I set read repair chance to 100% on critical column families and I write a tool to scan every CF, every column of every row. This strikes me as very silly.
Q1: Do I need to scan every column or is looking at one column enough to trigger a read repair ?
2) Can someone explain to me how the repair works such that I don't totally trash my cluster or spill into work week ?
Is there any improvement and clarity in 1.2 ? How about 2.0 ?