cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Russell Bradberry (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-10070) Automatic repair scheduling
Date Mon, 07 Dec 2015 17:28:11 GMT


Russell Bradberry commented on CASSANDRA-10070:

While it may intuitively seem like you want to kick-off a repair as soon as a node comes back
online, it can be very dangerous in a production environment. 

Starting the most resource intensive process on a node that is already problematic, in a cluster
that is already having issues can exacerbate the issue and lead to a longer outage, or degradation,
than anticipated.  

Network reliability is also another aspect of this.  Lets say you have 3 nodes, RF=3 and there
is a partition dividing node A and node B.  All nodes are still actually, up, but in this
case node A will start a repair on B and B will start a repair on A.  Now 2/3 of your cluster
is un-needly repairing which can cause serious performance problems, especially when running
a loaded cluster.

Other times you might not want a repair automatically started:

 - The cluster is in the middle of a rolling upgrade where streaming is broken between versions.
 - Heavily loaded clusters during normal operation (some users schedule repairs at night to
not affect performance during normal hours of operation)
 - Clusters where the read-consistency is high enough to account for the hints beyond the
window allowing the user to schedule the repair for a time that makes sense for their cluster
and use-case.

> Automatic repair scheduling
> ---------------------------
>                 Key: CASSANDRA-10070
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Marcus Olsson
>            Assignee: Marcus Olsson
>            Priority: Minor
>             Fix For: 3.x
> Scheduling and running repairs in a Cassandra cluster is most often a required task,
but this can both be hard for new users and it also requires a bit of manual configuration.
There are good tools out there that can be used to simplify things, but wouldn't this be a
good feature to have inside of Cassandra? To automatically schedule and run repairs, so that
when you start up your cluster it basically maintains itself in terms of normal anti-entropy,
with the possibility for manual configuration.

This message was sent by Atlassian JIRA

View raw message