cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Mehra <akhilme...@gmail.com>
Subject Re: Convert single node C* to cluster (rebalancing problem)
Date Mon, 12 Jun 2017 22:24:53 GMT
Great point John.

The OP should also note that data distribution also depends on your schema
and incoming data profile.

If your schema is not modelled correctly you can easily end up unevenly
distributed data.

Cheers,
Akhil

On Tue, Jun 13, 2017 at 3:36 AM, John Hughes <johnthughes@gmail.com> wrote:

> Is the OP expecting a perfect 50%/50% split? That, to my experience, is
> not going to happen, it is almost always shifted from a fraction of a
> percent to a couple percent.
>
> Datacenter: eu-west
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address        Load       Tokens       Owns (effective)  Host ID
>                         Rack
> UN  XX.XX.XX.XX    22.71 GiB  256          47.6%
> 57dafdde-2f62-467c-a8ff-c91e712f89c9  1c
> UN  XX.XX.XX.XX  17.17 GiB  256          51.3%
> d2a65c51-087d-48de-ae1f-a41142eb148d  1b
> UN  XX.XX.XX.XX  26.15 GiB  256          52.4%
> acf5dd34-5b81-4e5b-b7be-85a7fccd8e1c  1c
> UN  XX.XX.XX.XX   16.64 GiB  256          50.2%
> 6c8842dd-a966-467c-a7bc-bd6269ce3e7e  1a
> UN  XX.XX.XX.XX  24.39 GiB  256          49.8%
> fd92525d-edf2-4974-8bc5-a350a8831dfa  1a
> UN  XX.XX.XX.XX   23.8 GiB   256          48.7%
> bdc597c0-718c-4ef6-b3ef-7785110a9923  1b
>
> Though maybe part of what you are experiencing can be cleared up by
> repair/compaction/cleanup. Also, what are your outputs when you call out
> specific keyspaces? Do the numbers get more even?
>
> Cheers,
>
> On Mon, Jun 12, 2017 at 5:22 AM Akhil Mehra <akhilmehra@gmail.com> wrote:
>
>> auto_bootstrap is true by default. Ensure its set to true. On startup
>> look at your logs for your auto_bootstrap value.  Look at the node
>> configuration line in your log file.
>>
>> Akhil
>>
>> On Mon, Jun 12, 2017 at 6:18 PM, Junaid Nasir <jnasir@an10.io> wrote:
>>
>>> No, I didn't set it (left it at default value)
>>>
>>> On Fri, Jun 9, 2017 at 3:18 AM, ZAIDI, ASAD A <az192g@att.com> wrote:
>>>
>>>> Did you make sure auto_bootstrap property is indeed set to [true] when
>>>> you added the node?
>>>>
>>>>
>>>>
>>>> *From:* Junaid Nasir [mailto:jnasir@an10.io]
>>>> *Sent:* Monday, June 05, 2017 6:29 AM
>>>> *To:* Akhil Mehra <akhilmehra@gmail.com>
>>>> *Cc:* Vladimir Yudovin <vladyu@winguzone.com>;
>>>> user@cassandra.apache.org
>>>> *Subject:* Re: Convert single node C* to cluster (rebalancing problem)
>>>>
>>>>
>>>>
>>>> not evenly, i have setup a new cluster with subset of data (around
>>>> 5gb). using the configuration above I am getting these results
>>>>
>>>>
>>>>
>>>> Datacenter: datacenter1
>>>>
>>>> =======================
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> --  Address      Load       Tokens       Owns (effective)  Host ID     Rack
>>>>
>>>> UN  10.128.2.1   4.86 GiB   256          44.9%             e4427611-c247-42ee-9404-371e177f5f17
 rack1
>>>>
>>>> UN  10.128.2.10  725.03 MiB  256         55.1%             690d5620-99d3-4ae3-aebe-8f33af54a08b
 rack1
>>>>
>>>> is there anything else I can tweak/check to make the distribution even?
>>>>
>>>>
>>>>
>>>> On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <akhilmehra@gmail.com>
>>>> wrote:
>>>>
>>>> So now the data is evenly balanced in both nodes?
>>>>
>>>>
>>>>
>>>> Refer to the following documentation to get a better understanding of
>>>> the roc_address and the broadcast_rpc_address https://
>>>> www.instaclustr.com/demystifying-cassandras-broadcast_address/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_demystifying-2Dcassandras-2Dbroadcast-5Faddress_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=MaTA43pugg78xQNfaOQElhyvd8k7CjVqZPr3IWALdWI&e=>.
>>>> I am surprised that your node started up with rpc_broadcast_address
>>>> set as this is an unsupported property. I am assuming you are using
>>>> Cassandra version 3.10.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Akhil
>>>>
>>>>
>>>>
>>>> On 2/06/2017, at 11:06 PM, Junaid Nasir <jnasir@an10.io> wrote:
>>>>
>>>>
>>>>
>>>> I am able to get it working. I added a new node with following changes
>>>>
>>>> #rpc_address:0.0.0.0
>>>>
>>>> rpc_address: 10.128.1.11
>>>>
>>>> #rpc_broadcast_address:10.128.1.11
>>>>
>>>> rpc_address was set to 0.0.0.0, (I ran into a problem previously
>>>> regarding remote connection and made these changes
>>>> https://stackoverflow.com/questions/12236898/apache-
>>>> cassandra-remote-access
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_12236898_apache-2Dcassandra-2Dremote-2Daccess&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=oj8BCLiyBDqqVQNqfGohFeujtqlzAkd-uwS878d4mg4&e=>
>>>> )
>>>>
>>>>
>>>>
>>>> should it be happening?
>>>>
>>>>
>>>>
>>>> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vladyu@winguzone.com>
>>>> wrote:
>>>>
>>>> Did you run "nodetool cleanup" on first node after second was
>>>> bootstrapped? It should clean rows not belonging to node after tokens
>>>> changed.
>>>>
>>>>
>>>>
>>>> Best regards, Vladimir Yudovin,
>>>>
>>>> *Winguzone
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com_-3Ffrom-3Dlist&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=Q1M5YRAsw0iUQKOIulEmO72RhdENQCRhpqZSjgxxHos&e=>
>>>> - Cloud Cassandra Hosting*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jnasir@an10.io
>>>> <jnasir@an10.io>>* wrote ----
>>>>
>>>>
>>>>
>>>> Cassandra ensure that adding or removing nodes are very easy and that
>>>> load is balanced between nodes when a change is made. but it's not working
>>>> in my case.
>>>>
>>>> I have a single node C* deployment (with 270 GB of data) and want to
>>>> load balance the data on multiple nodes, I followed this guide
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_operations_ops-5Fadd-5Fnode-5Fto-5Fcluster-5Ft.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=57WqcUduTb1GA2Ij5E1fXgw3Cf21HYBK_4l2HVryPrk&s=xnLuX4qqSZxVuY6Gz2NXRdc6TG8J7at9kDhkhBnWpnM&e=>
>>>>
>>>>
>>>> `nodetool status` shows 2 nodes but load is not balanced between them
>>>>
>>>> Datacenter: dc1
>>>>
>>>> ===============
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> --  Address      Load       Tokens       Owns (effective)  Host ID    Rack
>>>>
>>>> UN  10.128.0.7   270.75 GiB  256          48.6%        1a3f6faa-4376-45a8-9c20-11480ae5664c
 rack1
>>>>
>>>> UN  10.128.0.14  414.36 KiB  256          51.4%        66a89fbf-08ba-4b5d-9f10-55d52a199b41
 rack1
>>>>
>>>> I also ran 'nodetool repair' on new node but result is same. any
>>>> pointers would be appreciated :)
>>>>
>>>>
>>>>
>>>> conf file of new node
>>>>
>>>> cluster_name: 'cluster1'
>>>>
>>>>  - seeds: "10.128.0.7"
>>>> num_tokens: 256
>>>>
>>>> endpoint_snitch: GossipingPropertyFileSnitch
>>>>
>>>> Thanks,
>>>>
>>>> Junaid
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>

Mime
View raw message