On Fri, Aug 2, 2013 at 12:46 PM, Philippe <watcherfr@gmail.com> wrote:
I have a cluster with 5 keyspaces. I would like to move one of the keyspaces to a separate cluster because it has very different usage patterns that can be optimized on different hardware.

What would be the best way to do that online ie. without interrupting reads & writes. The keyspace is about 350GB on each of the 3 nodes.

I don't know that I can assert that it is the "best" way but the "easiest" way is as follows :

1) create a new cluster with same tokens and replication factor and strategy as the old cluster (and same number of nodes)
2) create keyspace on new cluster
3) fork writes so that they go to both old and to new cluster
4) flush/snapshot and rsync on-disk SSTables from source nodes (can be done with source nodes live) to new nodes, BUT NOT INTO THE DATA DIRECTORY [1], renaming them by inflating the numeric part of their filenames by LARGE_NUMBER (f/e "1000") such that there is no chance of name collision with the new node's newly flushed SSTables.

for each new node, one at a time :
5) drain and stop new node
6) move SSTables into data directory on new node
8) start new node

and finally :
9) de-fork writes

In theory you could probably "get away" with not restarting the new nodes and using nodetool refresh, but it's more complicated to do so and there is a nonzero risk of problems.

If you can interrupt writes while serving reads from the old cluster, the process becomes significantly simpler. You just interrupt writes, flush, copy this files to the new cluster and start. You don't have to avoid name collision, because your new cluster is not taking write and is therefore not flushing SSTables.

[1] If you copy the files into the data directory in 4), especially without inflating the numeric part of their name, they may be overwritten silently by newly flushed ones.