cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Gupta <var...@uber.com>
Subject Re: Convert single node C* to cluster (rebalancing problem)
Date Fri, 16 Jun 2017 04:39:00 GMT
Akhil,

As per the blog, nodetool status shows data size for node1 even for token
ranges it does not own. Ain't this is bug in Cassandra?

Yes, on disk data will be present but it should be reflected in nodetool
status.

On Thu, Jun 15, 2017 at 6:17 PM, Akhil Mehra <akhilmehra@gmail.com> wrote:

> Hi,
>
> I put together a blog explaining possible reasons for an unbalanced
> Cassandra nodes.
>
> http://abiasforaction.net/unbalanced-cassandra-cluster/
>
> Let me know if you have any questions.
>
> Cheers,
> Akhil
>
>
> On Thu, Jun 15, 2017 at 5:54 PM, Affan Syed <asyed@an10.io> wrote:
>
>> John,
>>
>> I am a co-worker with Junaid -- he is out sick, so just wanted to confirm
>> that one of your shots in the dark is correct. This is a RF of 1x
>>
>> "CREATE KEYSPACE orion WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '1'}  AND durable_writes = true;"
>>
>> However, how does the RF affect the redistribution of key/data?
>>
>> Affan
>>
>> - Affan
>>
>> On Wed, Jun 14, 2017 at 1:16 AM, John Hughes <johnthughes@gmail.com>
>> wrote:
>>
>>> OP, I was just looking at your original numbers and I have some
>>> questions:
>>>
>>> 270GB on one node and 414KB on the other, but something close to 50/50
>>> on "Owns(effective)".
>>> What replication factor are your keyspaces set up with? 1x or 2x or ??
>>>
>>> I would say you are seeing 50/50 because the tokens are allocated
>>> 50/50(others on the list please correct what are for me really just
>>> assumptions!!!!), but I would hazard a guess that your replication factor
>>> is still 1x, so it isn't moving anything around. Or your keyspace
>>> rplication is incorrect and isn't being distributed(I have had issues with
>>> the AWSMultiRegionSnitch and not getting the region correct[us-east vs
>>> us-east-1). It doesn't throw an error, but it doesn't work very well either
>>> =)
>>>
>>> Can you do a 'describe keyspace XXX' and show the first line(the CREATE
>>> KEYSPACE line).
>>>
>>> Mind you, these are all just shots in the dark from here.
>>>
>>> Cheers,
>>>
>>>
>>> On Tue, Jun 13, 2017 at 3:13 AM Junaid Nasir <jnasir@an10.io> wrote:
>>>
>>>> Is the OP expecting a perfect 50%/50% split?
>>>>
>>>>
>>>> best result I got was 240gb/30gb split, which I think is not properly
>>>> balanced.
>>>>
>>>>
>>>>> Also, what are your outputs when you call out specific keyspaces? Do
>>>>> the numbers get more even?
>>>>
>>>>
>>>> i don't know what you mean by *call out specific key spaces?* can you
>>>> please explain that a bit.
>>>>
>>>>
>>>> If your schema is not modelled correctly you can easily end up unevenly
>>>>> distributed data.
>>>>
>>>>
>>>> I think that is the problem. initial 270gb data might not by modeled
>>>> correctly. I have run a lot of tests on 270gb data including downsizing it
>>>> to 5gb, they all resulted in same uneven distribution. I also tested a
>>>> dummy dataset of 2gb which was balanced evenly. coming from rdb, I didn't
>>>> give much thought to data modeling. can anyone please point me to some
>>>> resources regarding this problem.
>>>>
>>>> On Tue, Jun 13, 2017 at 3:24 AM, Akhil Mehra <akhilmehra@gmail.com>
>>>> wrote:
>>>>
>>>>> Great point John.
>>>>>
>>>>> The OP should also note that data distribution also depends on your
>>>>> schema and incoming data profile.
>>>>>
>>>>> If your schema is not modelled correctly you can easily end up
>>>>> unevenly distributed data.
>>>>>
>>>>> Cheers,
>>>>> Akhil
>>>>>
>>>>> On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthughes@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Is the OP expecting a perfect 50%/50% split? That, to my experience,
>>>>>> is not going to happen, it is almost always shifted from a fraction
of a
>>>>>> percent to a couple percent.
>>>>>>
>>>>>> Datacenter: eu-west
>>>>>> ===================
>>>>>> Status=Up/Down
>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>> --  Address        Load       Tokens       Owns (effective)  Host
ID
>>>>>>                               Rack
>>>>>> UN  XX.XX.XX.XX    22.71 GiB  256          47.6%
>>>>>> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
>>>>>> UN  XX.XX.XX.XX  17.17 GiB  256          51.3%
>>>>>> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
>>>>>> UN  XX.XX.XX.XX  26.15 GiB  256          52.4%
>>>>>> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
>>>>>> UN  XX.XX.XX.XX   16.64 GiB  256          50.2%
>>>>>> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
>>>>>> UN  XX.XX.XX.XX  24.39 GiB  256          49.8%
>>>>>> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
>>>>>> UN  XX.XX.XX.XX   23.8 GiB   256          48.7%
>>>>>> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>>>>>>
>>>>>> Though maybe part of what you are experiencing can be cleared up
by
>>>>>> repair/compaction/cleanup. Also, what are your outputs when you call
out
>>>>>> specific keyspaces? Do the numbers get more even?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilmehra@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> auto_bootstrap is true by default. Ensure its set to true. On
>>>>>>> startup look at your logs for your auto_bootstrap value.  Look
at the node
>>>>>>> configuration line in your log file.
>>>>>>>
>>>>>>> Akhil
>>>>>>>
>>>>>>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jnasir@an10.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> No, I didn't set it (left it at default value)
>>>>>>>>
>>>>>>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az192g@att.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Did you make sure auto_bootstrap property is indeed set
to [true]
>>>>>>>>> when you added the node?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* Junaid Nasir [mailto:jnasir@an10.io]
>>>>>>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>>>>>>> *To:* Akhil Mehra <akhilmehra@gmail.com>
>>>>>>>>> *Cc:* Vladimir Yudovin <vladyu@winguzone.com>;
>>>>>>>>> user@cassandra.apache.org
>>>>>>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing
>>>>>>>>> problem)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> not evenly, i have setup a new cluster with subset of
data (around
>>>>>>>>> 5gb). using the configuration above I am getting these
results
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Datacenter: datacenter1
>>>>>>>>>
>>>>>>>>> =======================
>>>>>>>>>
>>>>>>>>> Status=Up/Down
>>>>>>>>>
>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>>>
>>>>>>>>> --  Address      Load       Tokens       Owns (effective)
 Host ID     Rack
>>>>>>>>>
>>>>>>>>> UN  10.128.2.1   4.86 GiB   256          44.9%      
      e4427611-c247-42ee-9404-371e177f5f17  rack1
>>>>>>>>>
>>>>>>>>> UN  10.128.2.10  725.03 MiB  256         55.1%      
      690d5620-99d3-4ae3-aebe-8f33af54a08b  rack1
>>>>>>>>>
>>>>>>>>> is there anything else I can tweak/check to make the
distribution
>>>>>>>>> even?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilmehra@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> So now the data is evenly balanced in both nodes?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Refer to the following documentation to get a better
understanding
>>>>>>>>> of the roc_address and the broadcast_rpc_address https://
>>>>>>>>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI&e=>.
>>>>>>>>> I am surprised that your node started up with rpc_broadcast_address
>>>>>>>>> set as this is an unsupported property. I am assuming
you are
>>>>>>>>> using Cassandra version 3.10.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Akhil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jnasir@an10.io>
wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am able to get it working. I added a new node with
following
>>>>>>>>> changes
>>>>>>>>>
>>>>>>>>> #rpc_address:0.0.0.0
>>>>>>>>>
>>>>>>>>> rpc_address: 10.128.1.11
>>>>>>>>>
>>>>>>>>> #rpc_broadcast_address:10.128.1.11
>>>>>>>>>
>>>>>>>>> rpc_address was set to 0.0.0.0, (I ran into a problem
previously
>>>>>>>>> regarding remote connection and made these changes
>>>>>>>>> https://stackoverflow.com/questions/12236898/apache-cassandr
>>>>>>>>> a-remote-access
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4&e=>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> should it be happening?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <
>>>>>>>>> vladyu@winguzone.com> wrote:
>>>>>>>>>
>>>>>>>>> Did you run "nodetool cleanup" on first node after second
was
>>>>>>>>> bootstrapped? It should clean rows not belonging to node
after tokens
>>>>>>>>> changed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards, Vladimir Yudovin,
>>>>>>>>>
>>>>>>>>> *Winguzone
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos&e=>
>>>>>>>>> - Cloud Cassandra Hosting*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir
>>>>>>>>> <jnasir@an10.io <jnasir@an10.io>>* wrote
----
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cassandra ensure that adding or removing nodes are very
easy and
>>>>>>>>> that load is balanced between nodes when a change is
made. but it's not
>>>>>>>>> working in my case.
>>>>>>>>>
>>>>>>>>> I have a single node C* deployment (with 270 GB of data)
and want
>>>>>>>>> to load balance the data on multiple nodes, I followed
this guide
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM&e=>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> `nodetool status` shows 2 nodes but load is not balanced
between
>>>>>>>>> them
>>>>>>>>>
>>>>>>>>> Datacenter: dc1
>>>>>>>>>
>>>>>>>>> ===============
>>>>>>>>>
>>>>>>>>> Status=Up/Down
>>>>>>>>>
>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>>>>>>
>>>>>>>>> --  Address      Load       Tokens       Owns (effective)
 Host ID    Rack
>>>>>>>>>
>>>>>>>>> UN  10.128.0.7   270.75 GiB  256          48.6%     
  1a3f6faa-4376-45a8-9c20-11480ae5664c  rack1
>>>>>>>>>
>>>>>>>>> UN  10.128.0.14  414.36 KiB  256          51.4%     
  66a89fbf-08ba-4b5d-9f10-55d52a199b41  rack1
>>>>>>>>>
>>>>>>>>> I also ran 'nodetool repair' on new node but result is
same. any
>>>>>>>>> pointers would be appreciated :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> conf file of new node
>>>>>>>>>
>>>>>>>>> cluster_name: 'cluster1'
>>>>>>>>>
>>>>>>>>>  - seeds: "10.128.0.7"
>>>>>>>>> num_tokens: 256
>>>>>>>>>
>>>>>>>>> endpoint_snitch: GossipingPropertyFileSnitch
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Junaid
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>

Mime
View raw message