cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <>
Subject Re: Repair when a replica is Down
Date Tue, 19 Jan 2016 17:05:15 GMT
On Tue, Jan 19, 2016 at 10:44 AM, Anuj Wadehra <>

> Consider a scenario where I have a 20 node clsuter, RF=5, Read/Write
> Quorum, gc grace period=20. My cluster is fault tolerant and it can afford
> 2 node failure. Suddenly, one node goes down due to some hardware issue.
> Its 10 days since my node is down, none of the 19 nodes are being repaired
> and now its decision time. I am not sure how soon issue would be fixed may
> be 8 days before gc grace, so I shouldnt remove node early and add node
> back as it would cause unnecessary streaming. At the same time, if I dont
> remove the failed node, my entire system health would be in question and it
> would be a panic situation as no data got repaired in last 10 days and gc
> grace is approaching. I need sufficient time to repair 19 nodes.
> What looked like a fault tolerant system which can afford 2 node failure,
> required urgent attention and manual decision making when a single node
> went down. Why cant we just go ahead and repair remaining replicas if some
> replicas are down? If failed node comes up before gc grace period, we would
> run repair to fix inconsistencies and otheriwse we would discard data and
> bootstrap. I think that would be a really robust fault tolerant system.

That makes sense.  It seems like having the option to ignore down replicas
during repair could be at least somewhat helpful, although it may be tricky
to decide how this should interact with incremental repairs.  If there
isn't a jira ticket for this already, can you open one with the scenario

Tyler Hobbs
DataStax <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message