Let me be specific on lost data -> lost a replica , the other 2 nodes have replicas
I am running read/write at quorum. At this point I have turned off my clients from talking to this node. So if that is the case I can potentially just nodetool repair (without changing IP). But would it be better if I copied over the data/mykeyspace from another replica and then run repair?
> ok, so we just lost the data on that node. are building the raid on it, butYou're saying the raid failed and data is gone?
> once it is up what is the best way to bring it back in the cluster
If data is gone, the safe way is to have it re-join the cluster:
> just let it come up and run nodetool repair
> copy data from another node and then run nodetool repair,
> do I still need to run repair immeidately if I copy the data? Want to
> schedule repair for later during non peak hours?
But note that in your case, since you've lost data (if I understand
you), it's effectively a completely new node. That means you either
want to switch it's IP address and go for the "recommended" approach,
or do the other option but that WILL mean the node is serving reads
with incorrect data, violating the consistency. Depending on your
application, this may or may not be the case.
Unless it's a major problem for you, I suggest bringing it back in
with a new IP address and make it be treated like a completely fresh
replacement node. Probably decreases the risk of mistakes happening.
As for the other stuff about repair in the e-mail you pasted; periodic
repairs are part of regular cluster maintenance. See:
/ Peter Schuller (@scode on twitter)