Removetoken should only be used when removing a dead node from a cluster, it's a much slower and more expensive operation since it triggers a repair so that the remaining nodes can figure out which data they should now have. Decommission on the other hand is much simpler, the node that's being decommissioned streams the data it has to the nodes that should have it, and then removes itself.

I don't know exactly what your load is like, but I think the best way to accomplish it is like this:
You have nodes: 1 2 3 4 5 6 7 8 9
Add SSD nodes: S1 1 2 3 S2 4 5 6 S3 7 8 9
Decommission 1, 4, 7
Check if you can remove more nodes
Decommission 2, 5, 8
Check if you can remove more nodes
Decommission 3, 6, 9
And when you've stopped, make sure your ring is balanced by using nodetool move.

It's probably a bad idea to run with a lopsided cluster where some servers are much faster than the others. If you have a replication factor of 3, that means that half of your data will be on two slow and one fast machines (so quorum will be slow) and the oher half will be on two fast and one slow machine (so quorum will be fast). This leads to the somewhat unintuitive conclusion that you can make the cluster go faster by removing nodes.

But it's your data and your cluster, so you need to measure and benchmark and figure out what's best for you and your app.


/Henrik

On Mon, Aug 27, 2012 at 4:22 AM, Mohit Anchlia <mohitanchlia@gmail.com> wrote:
use nodetool decommission and nodetool removetoken


On Sun, Aug 26, 2012 at 5:31 PM, Senthilvel Rangaswamy <senthilvel@gmail.com> wrote:
We have a cluster of 9 nodes in the ring. We would like SSD backed boxes. But we may not need 9
nodes in that case. What is the best way to downscale the cluster to 6 or 3 nodes.

--
..Senthil

"If there's anything more important than my ego around, I want it
 caught and shot now."
                                                    - Douglas Adams.