incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ikeda Anthony <anthony.ikeda....@gmail.com>
Subject Re: Local Quorum Performance...
Date Sun, 18 Sep 2011 04:23:12 GMT
I'm not sure if it's significant, but on first notice the IP addresses all have the same octets
in the ProperyFileSnitch, yet the EC2Snitch, all the octets are different.

Ergo:
PropertyFileSnitch states that are all in the same data centre [168] and the same rac [2].
EC2Snitch states that all nodes in 3 different data centres [20, 73, 236].

I'm still new at this too and may not have the full answer as we are prepping our prod env
with the PropertyFileSnitch 2DC's and 3 nodes per DC. Though our QA environment is configured
much the same way only it's 3 nodes in a single DC:

consistency: LOCAL_QUORUM
strategy: NetworkTopologyStrategy
strategy_options: datacenter1:3

Our distribution is 33% equally.

Just reading the docs on the datastax website I'm starting to wonder how the PropertyFileSnitch
distributes the data across the DC's:
For NetworkTopologyStrategy, it specifies the number of replicas per data center in a comma
separated list of datacenter_name:number_of_replicas. 

I'm wondering if you need to increase your replication factor to 3 to see the data replicate
across the DC's

Anthony


On 17/09/2011, at 8:36 PM, Chris Marino wrote:

> Anthony, We used the Ec2Snitch for one sets of runs, but for another set we're using
PropertyFileSnitch.
> 
> With the PropertyFileSnitch we see:
> 
> Address         DC          Rack        Status State   Load            Owns    Token
                                      
> 
>                                                                                85070591730234615865843651857942052865
     
> 
> 192.168.2.1     us-east     1b          Up     Normal  60.59 MB        50.00%  0    
                                      
> 
> 192.168.2.6     us-west     1c          Up     Normal  26.5 MB         0.00%   1    
                                      
> 
> 192.168.2.2     us-east     1b          Up     Normal  29.86 MB        50.00%  85070591730234615865843651857942052864
     
> 
> 192.168.2.7     us-west     1c          Up     Normal  60.63 MB        0.00%   85070591730234615865843651857942052865
  
> 
> 
> 
> While with the EC2Snitch wwe see:
> Address         DC          Rack        Status State   Load            Owns    Token
                                      
> 
>                                                                                85070591730234615865843651857942052865
     
> 
> 107.20.68.176   us-east     1b          Up     Normal  59.95 MB        50.00%  0    
                                      
> 
> 204.236.179.193 us-west     1c          Up     Normal  53.67 MB        0.00%   1    
                                      
> 
> 184.73.133.171  us-east     1b          Up     Normal  60.65 MB        50.00%  85070591730234615865843651857942052864
     
> 
> 204.236.166.4   us-west     1c          Up     Normal  26.33 MB        0.00%   85070591730234615865843651857942052865
    
> 
> 
> 
> What also strange is that the Load on the nodes changes as well. For example, node 204.236.166.4
sometimes is very low (~26KB), other times its closer to 30MB. We see the same kind of variability
in both clusters.
> 
> 
> For both clusters, we're running stress tests with the following options:
> 
> 
> --consistency-level=LOCAL_QUORUM --threads=4 --replication-strategy=NetworkTopologyStrategy
--strategy-properties=us-east:2,us-west:2 --column-size=128 --keep-going --num-keys=100000
-r
> 
> Any clues to what is going on here are greatly appreciated.
> 
> Thanks
> CM
> 
> On Sat, Sep 17, 2011 at 12:15 PM, Ikeda Anthony <anthony.ikeda.dev@gmail.com> wrote:
> What snitch do you have configured? We typically see a proper spread of data across all
our nodes equally.
> 
> Anthony
> 
> 
> On 17/09/2011, at 10:06 AM, Chris Marino wrote:
> 
>> Hi, I have a question about what to expect when running a cluster across datacenters
with Local Quorum consistency.
>> 
>> My simplistic assumption is that the performance of an 8 node cluster split across
2 data centers and running with local quorum would perform roughly the same as a 4 node cluster
in one data center.
>> 
>> I'm 95% certain we've set up the keyspace so that the entire range is in one datacenter
and the client is local. I see the keyspace split across all the local nodes, with remote
nodes owning 0%. Yet when I run the stress tests against this configuration with local quorum,
I see dramatically different results from when I ran the same tests against a 4 node cluster.
 I'm still 5% unsure of this because the documentation on how to configure this is pretty
thin.
>> 
>> My understanding of Local Quorum was that once the data was written to a local quorum,
the commit would complete. I also believed that this would eliminate any WAN latency required
for replication to the other DC.
>> 
>> It not just that the split cluster runs slower, its also that there is enormous variability
in identical tests. Sometimes by a factor of 2 or more. It seems as though the WAN latency
is not only impacting performance, but that it's also introducing a wide variation on overally
performance.
>> 
>> Should WAN latency be completely hidden with local quorum? Or are there second order
issues involved that will impact performance??
>> 
>> I'm running in EC2 across us-east/west regions. I already know how unpredictable
EC2 performance can be, but what I'm seeing with here is far beyond normal.performance variability
for EC2
>> 
>> Is there something obvious that I'm missing that would explain why the results are
so different?? 
>> 
>> Here's the config when we run a 2x2 cluster:
>> 
>> Address         DC          Rack        Status State   Load            Owns    Token
                                      
>>                                                                                85070591730234615865843651857942052865
     
>> 192.168.2.1     us-east     1b          Up     Normal  25.26 MB        50.00%  0
                                          
>> 192.168.2.6     us-west     1c          Up     Normal  12.68 MB        0.00%   1
                                          
>> 192.168.2.2     us-east     1b          Up     Normal  12.56 MB        50.00%  85070591730234615865843651857942052864
     
>> 192.168.2.7     us-west     1c          Up     Normal  25.48 MB        0.00%   85070591730234615865843651857942052865
     
>> 
>> Thanks in advance.
>> CM
> 
> 


Mime
View raw message