incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Disable reads during node rebuild
Date Mon, 12 May 2014 21:58:36 GMT
> I'm not able to replace a dead node using the ordinary procedure (boostrap+join), and
would like to rebuild the replacement node from another DC.
Normally when you want to add a new DC to the cluster the command to use is nodetool rebuild
$DC_NAME .(with auto_bootstrap: false) That will get the node to stream data from the $DC_NAME

> The problem is that if I start a node with auto_bootstrap=false to perform the rebuild,
it automatically starts serving empty reads (CL=LOCAL_ONE).

When adding a new DC the nodes wont be processing reads, that is not the case for you.

You should disable the client API’s to prevent the clients from calling the new nodes, use
-Dcassandra.start_rpc=false and -Dcassandra.start_native_transport=false in cassandra-env.sh
or appropriate settings in cassandra.yaml

Disabling reads from other nodes will be harder. IIRC during bootstrap a different timeout
(based on ring_delay) is used to detect if the bootstrapping node is down. However if the
node is running and you use nodetool rebuild i’m pretty sure the normal gossip failure detectors
will kick in. Which means you cannot disable gossip to prevent reads. Also we would want the
node to be up for writes. 

But what you can do is artificially set the severity of the node high so the dynamic snitch
will route around it. See https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37


* Set the value to something high on the node you will be rebuilding, the number or cores
on the system should do.  (jmxterm is handy for this http://wiki.cyclopsgroup.org/jmxterm)

* Check nodetool gossipinfo on the other nodes to see the SEVERITY app state has propagated.

* Watch completed ReadStage tasks on the node you want to rebuild. If you have read repair
enabled it will still get some traffic. 
* Do rebuild 
* Reset severity to 0

Hope that helps. 
Aaron

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes <paulo.motta@chaordicsystems.com>
wrote:

> Hello,
> 
> I'm not able to replace a dead node using the ordinary procedure (boostrap+join), and
would like to rebuild the replacement node from another DC. The problem is that if I start
a node with auto_bootstrap=false to perform the rebuild, it automatically starts serving empty
reads (CL=LOCAL_ONE).
> 
> Is there a way to disable reads from a node while performing rebuild from another datacenter?
I tried starting the node in write survery mode, but the nodetool rebuild command does not
work in this mode.
> 
> Thanks,
> 
> -- 
> Paulo Motta
> 
> Chaordic | Platform
> www.chaordic.com.br
> +55 48 3232.3200


Mime
View raw message