incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Baskar Duraikannu <baskar.duraika...@outlook.com>
Subject RE: Read repair
Date Tue, 29 Oct 2013 11:32:11 GMT
Aaron

Rack1 goes down and some writes happen in quorum against rack 2 and 3. Hinted handoff is set
to 30 mins. After couple of hours rack1 comes back and rack2 goes down. Hinted handoff will
play but will not cover all of the writes because of 30 min setting. Now for rows inserted
for about 1 hour and 30 mins, there is no quorum until failed rack comes back up.

Hope this explains the scenario.
________________________________
From: Aaron Morton<mailto:aaron@thelastpickle.com>
Sent: ‎10/‎28/‎2013 2:42 AM
To: Cassandra User<mailto:user@cassandra.apache.org>
Subject: Re: Read repair

> As soon as it came back up, due to some human error, rack1 goes down. Now for some rows
it is possible that Quorum cannot be established.
Not sure I follow here.

if the first rack has come up I assume all nodes are available, if you then lose a different
rack I assume you have 2/3 of the nodes available and would be able to achieve a QUORUM.

> Just to minimize the issues, we are thinking of running read repair manually every night.
If you are reading and writing at QUORUM and the cluster does not have a QUORUM of nodes available
writes will not be processed. During reads any mismatch between the data returned from the
nodes will be detected and resolved before returning to the client.

Read Repair is an automatic process that reads from more nodes than necessary and resolves
the differences in the back ground.

I would run nodetool repair / Anti Entropy as normal, once on every machine every gc_grace_seconds.
If you have a while rack fail for run repair on the nodes in the rack if you want to get it
back to consistency quickly. The need to do that depends on the config for Hinted Handoff,
read_repair_chance, Consistency level, the write load, and (to some degree) the number of
nodes. If you want to be extra safe just run it.

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/10/2013, at 2:54 pm, Baskar Duraikannu <baskar.duraikannu@outlook.com> wrote:

> We are thinking through the deployment architecture for our Cassandra cluster.  Let us
say that we choose to deploy data across three racks.
>
> If let us say that one rack power went down for 10 mins and then it came back. As soon
as it came back up, due to some human error, rack1 goes down. Now for some rows it is possible
that Quorum cannot be established. Just to minimize the issues, we are thinking of running
read repair manually every night.
>
> Is this a good idea? How often do you perform read repair on your cluster?
>


Mime
View raw message