incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francisco Nogueira Calmon Sobral <fsob...@igcorp.com.br>
Subject Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Date Mon, 11 Nov 2013 14:05:57 GMT
We also run into the same problem when trying to load data from a 8 node cluster (C* 1.2.1,
Vnodes and same rack) into a 9 node cluster (C* 1.2.11, Vnodes and different racks) using
sstableloader.

We observed that a Key that clearly belonged to 3 nodes (using nodetool getendpoints) has
not been sent to any of them. Why?

Best regards,
Francisco.



On Sep 24, 2013, at 7:12 PM, Suruchi Deodhar <suruchi.deodhar@generalsentiment.com>
wrote:

> As an update to this thread, we conducted several tests with Cassandra-1.2.9, varying
parameters such as partitioner (Murmur3Partitioner/RandomParttioner), using NetworkToplogyStrategy
(with Ec2Snitch) / SimpleStrategy (with SimpleSnitch) across 2 Availability zones and 1 AZ.
We also tested the configurations separately with vnodes and without vnodes. 
> 
> Every time before each test, we wiped the cassandra cluster data and commitlog folders
and restarted with an empty cassandra db. However, in all the cases using 1.2.9 we continued
to see very heavy imbalance across the nodes as reported in this thread. 
> 
> We then tested the same exports with cassandra 1.2.5 version that we had been testing
previously (without vnodes across 2 AZs) and the data was balanced across the nodes of the
cluster. The output from bin/nodetool status is attached.
> 
> Was there some change from 1.2.5 to 1.2.9 that could be responsible for the imbalance
or is there some parameter setting that we may have completely missed in our configuration
wrt 1.2.9? Has anyone else experienced such an imbalance issue?
> 
> Also,  we were contemplating on using vnodes with NetworkTopologyStrategy (We want to
replicate data across 2 AZs)
> We came across the following links that mention that vnodes with NetworkToplogyStrategy
may create hotspots and the issue is marked as Open. Does that mean using vnodes with NetworkToplogyStrategy
is a bad idea?
> 
> [ https://issues.apache.org/jira/browse/CASSANDRA-4658 , https://issues.apache.org/jira/browse/CASSANDRA-3810
, https://issues.apache.org/jira/browse/CASSANDRA-4123 ] .
> 
> Thanks again for all your replies. 
> 
> Suruchi
> 
>   
> 
> 
> 
> On Fri, Sep 20, 2013 at 7:04 PM, Robert Coli <rcoli@eventbrite.com> wrote:
> On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <suruchi.deodhar@generalsentiment.com>
wrote:
> Using the nodes in the same availability zone(us-east-1b), we still get a highly imbalanced
cluster. The nodetool status and ring output is attached. Even after running repairs, the
cluster does not seem to balance.
> 
> If your cluster doesn't experience exceptions when loading and/or store a lot of hints,
repair is almost certainly just wasting your and your CPU's time.
> 
> =Rob 
> 
> <status_1.2.5_wo_vnodes_lz4.txt>


Mime
View raw message