incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Anchlia <mohitanch...@gmail.com>
Subject Re: Cassandra 1.2.9 cluster with vnodes is heavily unbalanced.
Date Fri, 20 Sep 2013 16:37:05 GMT
Like I said in my previous reply that I am not sure if that is the
problem and that's why I thought it would be a good test to do your test
with cluster in one RACK only.

I'll take a look at your ring output today. Did you also post cfstats
output?

On Fri, Sep 20, 2013 at 9:24 AM, Jayadev Jayaraman <jdisalive@gmail.com>wrote:

> As a follow-up, is operating a Cassandra cluster with machines on multiple
> racks and vnodes bound to cause load imbalance ? Shouldn't token-ranges
> assigned to individual machines via their vnodes be approximately balanced
> ? We're otherwise unable to explain why this imbalance occurs. ( it
> shouldn't be the fault of the Murmur3 partitioner which guarantees a
> uniform distribution of keys across token-ranges according to the doc. )
>
>
> On Thu, Sep 19, 2013 at 8:56 PM, Jayadev Jayaraman <jdisalive@gmail.com>wrote:
>
>> We're using NetworkTopologyStrategy for our placement_strategy and our
>> primary motivation for placing nodes on more than 1 rack is to prevent
>> showstoppers caused by localized faults leading to multiple EC2 instances
>> in the same availability zone (rack) becoming unreachable (which we have
>> encountered before).
>>
>> I've attached the output of "nodetool ring" here.
>>
>>
>> On Thu, Sep 19, 2013 at 8:35 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:
>>
>>> Other thing I noticed is that you are using mutiple RACKS and that might
>>> be contributing factor to it. However, I am not sure.
>>>
>>> Can you paste the output of nodetool cfstats and ring? Is it possible to
>>> run the same test but keeping all the nodes in one rack?
>>>
>>> I think you should open a JIRA if you are able to reproduce this.
>>>
>>> On Thu, Sep 19, 2013 at 4:41 PM, Jayadev Jayaraman <jdisalive@gmail.com>wrote:
>>>
>>>> We ran nodetool repair on all nodes for all Keyspaces / CFs, restarted
>>>> cassandra and this is what we get for nodetool status :
>>>>
>>>> bin/nodetool -h localhost status
>>>> Datacenter: us-east
>>>> ===================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> --  Address         Load       Tokens  Owns (effective)  Host ID
>>>>                         Rack
>>>> UN  10.238.133.174  885.36 MB  256     8.4%
>>>>  e41d8863-ce37-4d5c-a428-bfacea432a35  1a
>>>> UN  10.238.133.97   468.66 MB  256     7.7%
>>>>  1bf42b5e-4aed-4b06-bdb3-65a78823b547  1a
>>>> UN  10.151.86.146   1.08 GB    256     8.0%
>>>>  8952645d-4a27-4670-afb2-65061c205734  1a
>>>> UN  10.138.10.9     941.44 MB  256     8.6%
>>>>  25ccea82-49d2-43d9-830c-b9c9cee026ec  1a
>>>> UN  10.87.87.240    99.69 MB   256     8.6%
>>>>  ea066827-83bc-458c-83e8-bd15b7fc783c  1b
>>>> UN  10.93.5.157     87.44 MB   256     7.6%
>>>>  4ab9111c-39b4-4d15-9401-359d9d853c16  1b
>>>> UN  10.238.137.250  561.42 MB  256     7.8%
>>>>  84301648-afff-4f06-aa0b-4be421e0d08f  1a
>>>> UN  10.92.231.170   893.75 MB  256     9.3%
>>>>  a18ce761-88a0-4407-bbd1-c867c4fecd1f  1b
>>>> UN  10.138.2.20     31.89 MB   256     7.9%
>>>>  a6d4672a-0915-4c64-ba47-9f190abbf951  1a
>>>> UN  10.93.31.44     312.52 MB  256     7.8%
>>>>  67a6c0a6-e89f-4f3e-b996-cdded1b94faf  1b
>>>> UN  10.93.91.139    30.46 MB   256     8.1%
>>>>  682dd848-7c7f-4ddb-a960-119cf6491aa1  1b
>>>> UN  10.236.138.169  260.15 MB  256     9.1%
>>>>  cbbf27b0-b53a-4530-bfdf-3764730b89d8  1a
>>>> UN  10.137.7.90     38.45 MB   256     7.4%
>>>>  17b79aa7-64fc-4e16-b96a-955b0aae9bb4  1a
>>>> UN  10.93.77.166    867.15 MB  256     8.8%
>>>>  9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb  1b
>>>> UN  10.120.249.140  863.98 MB  256     9.4%
>>>>  e1fb69b0-8e66-4deb-9e72-f901d7a14e8a  1b
>>>> UN  10.90.246.128   242.63 MB  256     8.4%
>>>>  054911ec-969d-43d9-aea1-db445706e4d2  1b
>>>> UN  10.123.95.248   171.51 MB  256     7.2%
>>>>  a17deca1-9644-4520-9e62-ac66fc6fef60  1b
>>>> UN  10.136.11.40    33.8 MB    256     8.5%
>>>>  66be1173-b822-40b5-b650-cb38ae3c7a51  1a
>>>> UN  10.87.90.42     38.01 MB   256     8.0%
>>>>  dac0c6ea-56c6-44da-a4ec-6388f39ecba1  1b
>>>> UN  10.87.75.147    579.29 MB  256     8.3%
>>>>  ac060edf-dc48-44cf-a1b5-83c7a465f3c8  1b
>>>> UN  10.151.49.88    151.06 MB  256     8.9%
>>>>  57043573-ab1b-4e3c-8044-58376f7ce08f  1a
>>>> UN  10.87.83.107    512.91 MB  256     8.3%
>>>>  0019439b-9f8a-4965-91b8-7108bbb55593  1b
>>>> UN  10.238.170.159  85.04 MB   256     9.4%
>>>>  32ce322e-4f7c-46c7-a8ce-bd73cdd54684  1a
>>>> UN  10.137.20.183   167.41 MB  256     8.4%
>>>>  15951592-8ab2-473d-920a-da6e9d99507d  1a
>>>>
>>>> It doesn't seem to have changed by much. The loads are still highly
>>>> uneven.
>>>>
>>>> As for the number of keys in each node's CFs : the largest node now
>>>> has 5589120 keys for the column-family that had 6527744 keys before
>>>> (load is now 1.08 GB as compares to 1.05 GB before), while the smallest
>>>> node now has 71808 keys as compared to 3840 keys before (load is now
>>>> 31.89 MB as compares to 1.12 MB before).
>>>>
>>>>
>>>> On Thu, Sep 19, 2013 at 5:18 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:
>>>>
>>>>> Can you run nodetool repair on all the nodes first and look at the
>>>>> keys?
>>>>>
>>>>>
>>>>> On Thu, Sep 19, 2013 at 1:22 PM, Suruchi Deodhar <
>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>
>>>>>> Yes, the key distribution does vary across the nodes. For example,
on
>>>>>> the node with the highest data, Number of Keys (estimate) is 6527744
for a
>>>>>> particular column family, whereas for the same column family on the
node
>>>>>> with least data, Number of Keys (estimate) = 3840.
>>>>>>
>>>>>> Is there a way to control this distribution by setting some parameter
>>>>>> of cassandra.
>>>>>>
>>>>>> I am using the Murmur3 partitioner with NetworkTopologyStrategy.
>>>>>>
>>>>>> Thanks,
>>>>>> Suruchi
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2013 at 3:59 PM, Mohit Anchlia <
>>>>>> mohitanchlia@gmail.com> wrote:
>>>>>>
>>>>>>> Can you check cfstats to see number of keys per node?
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 19, 2013 at 12:36 PM, Suruchi Deodhar <
>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>
>>>>>>>> Thanks for your replies. I wiped out my data from the cluster
and
>>>>>>>> also cleared the commitlog before restarting it with num_tokens=256.
I then
>>>>>>>> uploaded data using sstableloader.
>>>>>>>>
>>>>>>>> However, I am still not able to see a uniform distribution
of data
>>>>>>>> across nodes of the clusters.
>>>>>>>>
>>>>>>>> The output of the bin/nodetool -h localhost status commands
looks
>>>>>>>> like follows. Some nodes have data as low as 1.12MB while
some have as high
>>>>>>>> as 912.57 MB.
>>>>>>>>
>>>>>>>> Datacenter: us-east
>>>>>>>> ===================
>>>>>>>> Status=Up/Down
>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>> --  Address         Load       Tokens  Owns (effective) 
Host
>>>>>>>> ID                               Rack
>>>>>>>> UN  10.238.133.174  856.66 MB  256     8.4%
>>>>>>>> e41d8863-ce37-4d5c-a428-bfacea432a35  1a
>>>>>>>> UN  10.238.133.97   439.02 MB  256     7.7%
>>>>>>>> 1bf42b5e-4aed-4b06-bdb3-65a78823b547  1a
>>>>>>>> UN  10.151.86.146   1.05 GB    256     8.0%
>>>>>>>> 8952645d-4a27-4670-afb2-65061c205734  1a
>>>>>>>> UN  10.138.10.9     912.57 MB  256     8.6%
>>>>>>>> 25ccea82-49d2-43d9-830c-b9c9cee026ec  1a
>>>>>>>> UN  10.87.87.240    70.85 MB   256     8.6%
>>>>>>>> ea066827-83bc-458c-83e8-bd15b7fc783c  1b
>>>>>>>> UN  10.93.5.157     60.56 MB   256     7.6%
>>>>>>>> 4ab9111c-39b4-4d15-9401-359d9d853c16  1b
>>>>>>>> UN  10.92.231.170   866.73 MB  256     9.3%
>>>>>>>> a18ce761-88a0-4407-bbd1-c867c4fecd1f  1b
>>>>>>>> UN  10.238.137.250  533.77 MB  256     7.8%
>>>>>>>> 84301648-afff-4f06-aa0b-4be421e0d08f  1a
>>>>>>>> UN  10.93.91.139    478.45 KB  256     8.1%
>>>>>>>> 682dd848-7c7f-4ddb-a960-119cf6491aa1  1b
>>>>>>>> UN  10.138.2.20     1.12 MB    256     7.9%
>>>>>>>> a6d4672a-0915-4c64-ba47-9f190abbf951  1a
>>>>>>>> UN  10.93.31.44     282.65 MB  256     7.8%
>>>>>>>> 67a6c0a6-e89f-4f3e-b996-cdded1b94faf  1b
>>>>>>>> UN  10.236.138.169  223.66 MB  256     9.1%
>>>>>>>> cbbf27b0-b53a-4530-bfdf-3764730b89d8  1a
>>>>>>>> UN  10.137.7.90     11.36 MB   256     7.4%
>>>>>>>> 17b79aa7-64fc-4e16-b96a-955b0aae9bb4  1a
>>>>>>>> UN  10.93.77.166    837.64 MB  256     8.8%
>>>>>>>> 9a821d1e-40e5-445d-b6b7-3cdd58bdb8cb  1b
>>>>>>>> UN  10.120.249.140  838.59 MB  256     9.4%
>>>>>>>> e1fb69b0-8e66-4deb-9e72-f901d7a14e8a  1b
>>>>>>>> UN  10.90.246.128   216.75 MB  256     8.4%
>>>>>>>> 054911ec-969d-43d9-aea1-db445706e4d2  1b
>>>>>>>> UN  10.123.95.248   147.1 MB   256     7.2%
>>>>>>>> a17deca1-9644-4520-9e62-ac66fc6fef60  1b
>>>>>>>> UN  10.136.11.40    4.24 MB    256     8.5%
>>>>>>>> 66be1173-b822-40b5-b650-cb38ae3c7a51  1a
>>>>>>>> UN  10.87.90.42     11.56 MB   256     8.0%
>>>>>>>> dac0c6ea-56c6-44da-a4ec-6388f39ecba1  1b
>>>>>>>> UN  10.87.75.147    549 MB     256     8.3%
>>>>>>>> ac060edf-dc48-44cf-a1b5-83c7a465f3c8  1b
>>>>>>>> UN  10.151.49.88    119.86 MB  256     8.9%
>>>>>>>> 57043573-ab1b-4e3c-8044-58376f7ce08f  1a
>>>>>>>> UN  10.87.83.107    484.3 MB   256     8.3%
>>>>>>>> 0019439b-9f8a-4965-91b8-7108bbb55593  1b
>>>>>>>> UN  10.137.20.183   137.67 MB  256     8.4%
>>>>>>>> 15951592-8ab2-473d-920a-da6e9d99507d  1a
>>>>>>>> UN  10.238.170.159  49.17 MB   256     9.4%
>>>>>>>> 32ce322e-4f7c-46c7-a8ce-bd73cdd54684  1a
>>>>>>>>
>>>>>>>> Is there something else that I should be doing differently?
>>>>>>>>
>>>>>>>> Thanks for your help!
>>>>>>>>
>>>>>>>> Suruchi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 19, 2013 at 3:20 PM, Richard Low <richard@wentnet.com>wrote:
>>>>>>>>
>>>>>>>>> The only thing you need to guarantee is that Cassandra
doesn't
>>>>>>>>> start with num_tokens=1 (the default in 1.2.x) or, if
it does, that you
>>>>>>>>> wipe all the data before starting it with higher num_tokens.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19 September 2013 19:07, Robert Coli <rcoli@eventbrite.com>wrote:
>>>>>>>>>
>>>>>>>>>> On Thu, Sep 19, 2013 at 10:59 AM, Suruchi Deodhar
<
>>>>>>>>>> suruchi.deodhar@generalsentiment.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Do you suggest I should try with some other installation
>>>>>>>>>>> mechanism? Are there any known problems with
the tar installation of
>>>>>>>>>>> cassandra 1.2.9 that I should be aware of?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I was asking in the context of this JIRA :
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-2356
>>>>>>>>>>
>>>>>>>>>> Which does not seem to apply in your case!
>>>>>>>>>>
>>>>>>>>>> =Rob
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message