incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Stephen.M.Thomp...@wellsfargo.com>
Subject RE: unbalanced ring
Date Wed, 06 Feb 2013 14:02:24 GMT
Thanks Aaron.  I ran the cassandra-shuffle job and did a rebuild and compact on each of the
nodes.

[root@Config3482VM1 apache-cassandra-1.2.1]# bin/nodetool status
Datacenter: 28
==============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns (effective)  Host ID                       
       Rack
UN  10.28.205.125     1.7 GB     255     33.7%             3daab184-61f0-49a0-b076-863f10bc8c6c
 205
UN  10.28.205.126     591.44 MB  256     99.9%             55bbd4b1-8036-4e32-b975-c073a7f0f47f
 205
UN  10.28.205.127     112.28 MB  257     66.4%             d240c91f-4901-40ad-bd66-d374a0ccf0b9
 205

So this is a little better.  At last node 3 has some content, but they are still far from
balanced.  If I am understand this correctly, this is the distribution I would expect if the
tokens were set at 15/5/1 rather than equal.  As configured, I would expect roughly equal
amounts of data on each node. Is that right?  Do you have any suggestions for what I can look
at to get there?

I have about 11M rows of data in this keyspace and none of them are exceptionally long ...
it's data pulled from Oracle and didn't include any BLOB, etc.

Stephen Thompson
Wells Fargo Corporation
Internet Authentication & Fraud Prevention
704.427.3137 (W) | 704.807.3431 (C)

This message may contain confidential and/or privileged information, and is intended for the
use of the addressee only. If you are not the addressee or authorized to receive this for
the addressee, you must not use, copy, disclose, or take any action based on this message
or any information herein. If you have received this message in error, please advise the sender
immediately by reply e-mail and delete this message. Thank you for your cooperation.

From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, February 05, 2013 3:41 PM
To: user@cassandra.apache.org
Subject: Re: unbalanced ring

Use nodetool status with vnodes http://www.datastax.com/dev/blog/upgrading-an-existing-cluster-to-vnodes

The different load can be caused by rack affinity, are all the nodes in the same rack ? Another
simple check is have you created some very big rows?
Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 6/02/2013, at 8:40 AM, Stephen.M.Thompson@wellsfargo.com<mailto:Stephen.M.Thompson@wellsfargo.com>
wrote:


So I have three nodes in a ring in one data center.  My configuration has num_tokens: 256
set andinitial_token commented out.  When I look at the ring, it shows me all of the token
ranges of course, and basically identical data for each range on each node.  Here is the Cliff's
Notes version of what I see:

[root@Config3482VM2 apache-cassandra-1.2.0]# bin/nodetool ring

Datacenter: 28
==========
Replicas: 1

Address         Rack        Status State   Load            Owns                Token
                                                                               9187343239835811839
10.28.205.125   205         Up     Normal  2.85 GB         33.69%              -3026347817059713363
10.28.205.125   205         Up     Normal  2.85 GB         33.69%              -3026276684526453414
10.28.205.125   205         Up     Normal  2.85 GB         33.69%              -3026205551993193465
  (etc)
10.28.205.126   205         Up     Normal  1.15 GB         100.00%             -9187343239835811840
10.28.205.126   205         Up     Normal  1.15 GB         100.00%             -9151314442816847872
10.28.205.126   205         Up     Normal  1.15 GB         100.00%             -9115285645797883904
  (etc)
10.28.205.127   205         Up     Normal  69.13 KB        66.30%              -9223372036854775808
10.28.205.127   205         Up     Normal  69.13 KB        66.30%              36028797018963967
10.28.205.127   205         Up     Normal  69.13 KB        66.30%              72057594037927935
  (etc)

So at this point I have a number of questions.   The biggest question is of Load.  Why does
the .125 node have 2.85 GB, .126 has 1.15 GB, and .127 has only 0.000069 GB?  These boxes
are all comparable and all configured identically.

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

I'm sorry to ask so many questions - I'm having a hard time finding documentation that explains
this stuff.

Stephen


Mime
View raw message