Thanks for the explanation. The objective is not to have a perfectly balanced US-East and SG DC clusters. SG DC cluster is just a backup cluster and hence has lesser nodes than US-East cluster. What we are trying to figure out is the imbalance between the 6 nodes within US-East itself. I'll try to correct the 6 nodes with US-East to proper racks and check.

In addition, as I mentioned earlier, do you see any issues with the dynamic snitch attribute score? I see that node has high score, but what value of dynamic_snitch_badness_threshold should I set so that other replicas can get the traffic? ( that node has >50% higher score than all other nodes )

On Fri, Nov 1, 2013 at 10:04 PM, Tyler Hobbs <tyler@datastax.com> wrote:

On Fri, Nov 1, 2013 at 5:07 AM, Rakesh Rajan <rakeshxp@gmail.com> wrote:

1) By alternating racks, do you mean to alternate racks between all nodes in a single DC v/s multiple DCs? AWS EastCoast has 4 AZs and Singapore has 2 AZs. So is the final solution something like this: 
ip11 - East Coast - m1.xlarge / us-east-1b         - Token: 0
ip21 - Singapore  - m1.xlarge / ap-southeast-1a - Token: 1001
ip12 - East Coast - m1.xlarge / us-east-1c         - Token: 28356863910078205288614550619314017621
ip13 - East Coast - m1.xlarge / us-east-1d         - Token: 56713727820156410577229101238628035241
ip22 - Singapore  - m1.xlarge / ap-southeast-1b - Token: 56713727820156410577229101238628036241
ip14 - East Coast - m1.xlarge / us-east-1a         - Token: 85070591730234615865843651857942052863
ip15 - East Coast - m1.xlarge / us-east-1b         - Token: 113427455640312821154458202477256070484
ip23 - Singapore  - m1.xlarge / ap-southeast-1a - Token: 113427455640312821154458202477256071484
ip16 - East Coast - m1.xlarge / us-east-1c         - Token: 141784319550391026443072753096570088105

Is this what you had suggested? 

That would be more balanced than your current setup, but it would still be unbalanced, especially the ap-southeast DC.  To have a perfectly balanced cluster with multiple racks, you need to a) have the same number of nodes on each rack, and b) alternate racks within each DC.  Your new layout would meet requirement (b), but not (a).  This is why I suggest using the same rack for all nodes.

Tyler Hobbs