incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Date Fri, 20 Sep 2013 23:00:36 GMT
Did you start out your cluster after wiping all the sstables and commit
logs?

On Fri, Sep 20, 2013 at 3:42 PM, Suruchi Deodhar <
suruchi.deodhar@generalsentiment.com> wrote:

> We have been trying to resolve this issue to find a stable configuration
> that can give us a balanced cluster with equally distributed data across
> the nodes.
>
> We tried running a smaller 12 node cluster with following parameters:
>
> placement_strategy = NetworkTopologyStrategy ( 12 nodes in us-east-1b )
> partitioner=Murmur3Partitioner
> compaction strategy = LeveledCompactionStrategy
> replication factor = 2
> snitch = EC2Snitch
> vnodes with num_tokens = 256 ( 256 tokens per node )
>
> Using the nodes in the same availability zone(us-east-1b), we still get a
> highly imbalanced cluster. The nodetool status and ring output is attached.
> Even after running repairs, the cluster does not seem to balance.
>
> Datacenter: us-east
>> ===================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address         Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  10.120.249.140  641.11 KB  256     16.7%
>> 90320ebc-cd03-4303-9f91-a8eb534ecaa4  1b
>> UN  10.90.246.128   1.04 MB    256     16.4%
>> 14352484-07c2-4cf0-9c12-e53d9e19f23b  1b
>> UN  10.123.95.248   4.21 GB    256     17.7%
>> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d  1b
>> UN  10.87.87.240    337.46 KB  256     16.6%
>> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53  1b
>> UN  10.93.5.157     4.21 GB    256     17.0%
>> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803  1b
>> UN  10.92.231.170   1021.72 KB  256     15.5%
>> e4835262-e27c-4da3-b45d-758a860f506e  1b
>> UN  10.93.31.44     653.58 KB  256     16.5%
>> 19d24eda-b9bf-42cc-8463-5be71d28da9d  1b
>> UN  10.93.91.139    298.81 KB  256     17.8%
>> 6d8d5879-4340-4d7c-960d-a12a80a9fac3  1b
>> UN  10.87.90.42     3.75 MB    256     16.8%
>> ff64cfc0-d614-48b9-8e96-aee99114c6ff  1b
>> UN  10.87.75.147    3.55 MB    256     16.6%
>> ec1f182d-bb01-4546-a523-2a42df158ee0  1b
>> UN  10.93.77.166    333.88 KB  256     16.0%
>> 0964902c-736d-411a-bed5-341dbc96813f  1b
>> UN  10.87.83.107    543.2 KB   256     16.4%
>> 924fa50c-d253-41e8-b2fb-d1be2942f2c8  1b
>>
>> After repairs- status
>>
>
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address         Load       Tokens  Owns (effective)  Host
> ID                               Rack
> UN  10.120.249.140  82.26 MB   256     16.7%
> 90320ebc-cd03-4303-9f91-a8eb534ecaa4  1b
> UN  10.90.246.128   97.61 MB   256     16.4%
> 14352484-07c2-4cf0-9c12-e53d9e19f23b  1b
> UN  10.123.95.248   4.25 GB    256     17.7%
> 3d520c2a-f9a0-480a-83d8-675ed3f46c7d  1b
> UN  10.87.87.240    95.05 MB   256     16.6%
> 11bfebe6-f40c-4d4e-a76e-f6287a27fa53  1b
> UN  10.93.5.157     4.25 GB    256     17.0%
> 5a20ca7b-37fa-4e5b-86d4-c47c80e3d803  1b
> UN  10.92.231.170   86.54 MB   256     15.5%
> e4835262-e27c-4da3-b45d-758a860f506e  1b
> UN  10.93.31.44     102.42 MB  256     16.5%
> 19d24eda-b9bf-42cc-8463-5be71d28da9d  1b
> UN  10.93.91.139    82.85 MB   256     17.8%
> 6d8d5879-4340-4d7c-960d-a12a80a9fac3  1b
> UN  10.87.90.42     134.76 MB  256     16.8%
> ff64cfc0-d614-48b9-8e96-aee99114c6ff  1b
> UN  10.87.75.147    101.15 MB  256     16.6%
> ec1f182d-bb01-4546-a523-2a42df158ee0  1b
> UN  10.93.77.166    73.66 MB   256     16.0%
> 0964902c-736d-411a-bed5-341dbc96813f  1b
> UN  10.87.83.107    93.73 MB   256     16.4%
> 924fa50c-d253-41e8-b2fb-d1be2942f2c8  1b
>
>
> So essentially, using NetworkTopology strategy even with a single AZ is
> not giving us a balanced cluster. Is there any alternative that we can try
> for a stable cluster? We want to use vnodes.
>
> Thanks,
> Suruchi
>
>
> On Fri, Sep 20, 2013 at 1:31 PM, Robert Coli <rcoli@eventbrite.com> wrote:
>
>> On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jdisalive@gmail.com>wrote:
>>
>>> As a follow-up, is operating a Cassandra cluster with machines on
>>> multiple racks and vnodes bound to cause load imbalance ? Shouldn't
>>> token-ranges assigned to individual machines via their vnodes be
>>> approximately balanced ? We're otherwise unable to explain why this
>>> imbalance occurs. ( it shouldn't be the fault of the Murmur3 partitioner
>>> which guarantees a uniform distribution of keys across token-ranges
>>> according to the doc. )
>>>
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-4658
>> https://issues.apache.org/jira/browse/CASSANDRA-4123
>>
>> tl;dr : Yes, vnodes makes the imbalance factor slightly greater, but in
>> practice it "shouldn't" be significant.
>>
>> If you are experiencing significant imbalance, one or both of those
>> tickets may be an appropriate forum to describe your experience.
>>
>> =Rob
>>
>>
>
>

Mime
View raw message