Thank you Edward.
I suspect that nodetool cleanup is IO intensive. So running nodetool cleanup concurrently on the entire cluster may have a significantly impact the IO performance of applications.
Apart from this, do you see any other implications on running the nodetool cleanup concurrently on the entire cluster ?
From: Edward Capriolo <firstname.lastname@example.org>
To: "email@example.com" <firstname.lastname@example.org>; Emalayan Vairavanathan <email@example.com>
Sent: Monday, 10 June 2013 2:53 PM
Subject: Re: [Cassandra] Expanding a Cassandra cluster
You eventually should run cleanup to remove data no longer needed on the node. However it does not need to be run quickly after a join. You can run it when you get around to it. I would run it on a few nodes at a time until they are all cleaned up.