Hi, I have a question about what to expect when running a cluster across datacenters with Local Quorum consistency.

My simplistic assumption is that the performance of an 8 node cluster split across 2 data centers and running with local quorum would perform roughly the same as a 4 node cluster in one data center.

I'm 95% certain we've set up the keyspace so that the entire range is in one datacenter and the client is local. I see the keyspace split across all the local nodes, with remote nodes owning 0%. Yet when I run the stress tests against this configuration with local quorum, I see dramatically different results from when I ran the same tests against a 4 node cluster.  I'm still 5% unsure of this because the documentation on how to configure this is pretty thin.

My understanding of Local Quorum was that once the data was written to a local quorum, the commit would complete. I also believed that this would eliminate any WAN latency required for replication to the other DC.

It not just that the split cluster runs slower, its also that there is enormous variability in identical tests. Sometimes by a factor of 2 or more. It seems as though the WAN latency is not only impacting performance, but that it's also introducing a wide variation on overally performance.

Should WAN latency be completely hidden with local quorum? Or are there second order issues involved that will impact performance??

I'm running in EC2 across us-east/west regions. I already know how unpredictable EC2 performance can be, but what I'm seeing with here is far beyond normal.performance variability for EC2

Is there something obvious that I'm missing that would explain why the results are so different?? 

Here's the config when we run a 2x2 cluster:

Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               85070591730234615865843651857942052865      
192.168.2.1     us-east     1b          Up     Normal  25.26 MB        50.00%  0                                           
192.168.2.6     us-west     1c          Up     Normal  12.68 MB        0.00%   1                                           
192.168.2.2     us-east     1b          Up     Normal  12.56 MB        50.00%  85070591730234615865843651857942052864      
192.168.2.7     us-west     1c          Up     Normal  25.48 MB        0.00%   85070591730234615865843651857942052865      

Thanks in advance.
CM