cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <>
Subject Re: How to determine if repair need to be run
Date Tue, 29 Mar 2011 17:44:09 GMT
> Yes but that doesn't really provide the monitoring that will really be
> helpful. If I don't realize it until 2 days then we potentially could be
> returning inconsistent results or not have data sync for 2 days until repair
> is run. It will be best to be able to monitor these things so that it can be
> run as soon as it is required (eg node down). Have such monitoring will be
> helpful for operations team to monitor also who may not know all internals
> of cassandra.

For the purpose of this discussion, nodes are always down in any
non-trivial time window. You may have flapping in the ring, individual
requests may time out, etc. Do not assume repair is not required just
because you have not had some kind of major outtage where a human
became consciously aware that a node was officially "down".

Unless you really know what you're doing, the thing to monitor is the
completion of repairs at sufficient frequency. In the event that
repair *doesn't* run, there needs to be enough time left until
tombstone expiry for someone to take some kind of action (whether that
is re-running repair again or re-configuring gcgraceseconds
temporarily is another matter).

Repair is not something that you only run in the event of some major
issue; repair is a regularly scheduled operation for your typical

The invariant required by Cassandra is that repairs complete prior to
tombstones expiring (see URL in previous e-mail). Some applications,
given some combination of consistency levels, use-case and
requirements, may benefit from more frequent repair. But the important
part, is the minimum repair frequency mandated by Cassandra - and
determined by GCGraceSeconds.

/ Peter Schuller

View raw message