incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayadev Jayaraman <jdisal...@gmail.com>
Subject Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Date Thu, 19 Sep 2013 01:06:13 GMT
Basic configuration :

version=1.2.9
partitioner=Murmur3Partitioner
compaction strategy = LeveledCompactionStrategy
Cluster = 24 m1.xlarge nodes on EC2 ( 1.7 TB ephemeral raid0 per node )
replication factor = 2
snitch = EC2Snitch
placement_strategy = NetworkTopologyStrategy ( 12 nodes each in zones
us-east-1a , us-east-1b )


We use vnodes with num_tokens = 256 ( 256 tokens per node ) . After loading
some data with sstableloader , we find that the cluster is heavily
imbalanced :

$ bin/nodetool -h localhost status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns (effective)  Host ID
                    Rack
UN  10.238.133.174  452.24 KB  256     8.1%
 f2dc02a5-f9a9-43b4-85d6-dbf3c87b044b  1a
UN  10.238.133.97   485.75 KB  256     5.9%
 b881777c-121d-4afb-adf5-0328386e901e  1a
UN  10.151.86.146   120.78 KB  256     0.5%
 34d83902-d4e3-4245-a20c-4e829bb4020a  1a
UN  10.138.10.9     355.19 KB  256     5.4%
 6f9868bf-fc87-4c0e-8e73-f02cf49f6ebe  1a
UN  10.87.87.240    393.25 KB  256     7.2%
 4eb6f0c9-f30a-4615-ab89-e86e0ab84bf9  1b
UN  10.93.5.157     3.39 MB    256     12.6%
4f913ba4-e318-4080-b76f-2ad262ad5c7a  1b
UN  10.238.137.250  3.81 MB    256     27.5%
7d0c8eca-d0ed-40dd-aa5f-2b798a8225d3  1a
UN  10.92.231.170   196.5 KB   256     2.2%
 4fe0ca5b-d58c-4376-8894-01ed4282bad8  1b
UN  10.93.31.44     162.36 KB  256     0.3%
 83e1a28d-1080-4efe-9110-7f2ee4ac8128  1b
UN  10.138.2.20     375.86 KB  256     8.5%
 a55f4480-3e17-4759-bcfc-04c04a71aa6b  1a
UN  10.93.91.139    760.45 KB  256     17.1%
ee373e42-63e4-4789-8532-cdd49dae5998  1b
UN  10.236.138.169  303.75 KB  256     5.5%
 1fd7e817-1d4b-4abd-89cf-956f295679e6  1a
UN  10.137.7.90     4.27 GB    256     18.4%
a8d26ff0-8d41-406e-b724-8ddef29511b2  1a
UN  10.93.77.166    1.32 MB    256     27.8%
7bc043a7-863b-4230-b4fb-e6765bafcda3  1b
UN  10.120.249.140  4.27 GB    256     7.8%
 10bb54e6-f597-4100-9408-070c45fdd449  1b
UN  10.123.95.248   608.91 KB  256     8.6%
 8ed80942-9fb9-4c2f-b951-5c7648998a8a  1b
UN  10.90.246.128   401.34 KB  256     6.0%
 485d3c01-e9b2-4172-9bbc-f0af86962570  1b
UN  10.136.11.40    447.6 KB   256     9.0%
 3adb2d0a-810c-475c-bb2f-89a15a53a234  1a
UN  10.87.90.42     241.55 KB  256     3.8%
 749730a2-121a-4d48-bef4-588c7df234ab  1b
UN  10.87.75.147    339.66 KB  256     4.6%
 031baedf-e22d-41a9-b196-af64b55129ba  1b
UN  10.151.49.88    438.69 KB  256     4.2%
 365d2dd3-d72c-4dd2-a833-db766e183fcc  1a
UN  10.87.83.107    170.49 KB  256     2.2%
 d42a015b-e3db-42a5-9895-66d915a3e4c7  1b
UN  10.238.170.159  491.37 KB  256     6.5%
 fd979cc7-d1d5-4c75-9608-6de861b0761b  1a
UN  10.137.20.183   163.06 KB  256     0.5%
 361749d8-6a37-4f65-91cd-70f3222a762e  1a

Some nodes claim to own 27.5% of the data, while some nodes claim less than
1% . Also, the loads on each node don't tally with the ownership
percentages ( e.g. 10.137.7.90 has a load of 4.27 GB and owns 18.4% while
10.238.137.250 has a load of 3.81 MB and owns 27.5% ) .

Can someone help me figure out if I'm missing out on something ? Even if
the data distribution isn't perfectly even, it definitely shouldn't be this
far off.

I've attached the output of the "nodetool ring" command in a file here if
it helps.

Mime
View raw message