cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Grasso <anthony.gra...@gmail.com>
Subject Re: Uneven token distribution with allocate_tokens_for_keyspace
Date Wed, 04 Dec 2019 23:59:32 GMT
Hi Enrico,

Glad to hear the problem has been resolved and thank you for the feedback!

Kind regards,
Anthony

On Mon, 2 Dec 2019 at 22:03, Enrico Cavallin <cavallin.enrico@gmail.com>
wrote:

> Hi Anthony,
> thank you for your hints, now the new DC is well balanced within 2%.
> I did read your article, but I thought it was needed only for new
> "clusters", not also for new "DCs"; but RF is per DC so it makes sense.
>
> You TLP guys are doing a great job for Cassandra community.
>
> Thank you,
> Enrico
>
>
> On Fri, 29 Nov 2019 at 05:09, Anthony Grasso <anthony.grasso@gmail.com>
> wrote:
>
>> Hi Enrico,
>>
>> This is a classic chicken and egg problem with the
>> allocate_tokens_for_keyspace setting.
>>
>> The allocate_tokens_for_keyspace setting uses the replication factor of
>> a DC keyspace to calculate the token allocation when a node is added to the
>> cluster for the first time.
>>
>> Nodes need to be added to the new DC before we can replicate the keyspace
>> over to it. Herein lies the problem. We are unable to use
>> allocate_tokens_for_keyspace unless the keyspace is replicated to the
>> new DC. In addition, as soon as you change the keyspace replication to the
>> new DC, new data will start to be written to it. To work around this issue
>> you will need to do the following.
>>
>>    1. Decommission all the nodes in the *dcNew*, one at a time.
>>    2. Once all the *dcNew* nodes are decommissioned, wipe the contents
>>    in the *commitlog*, *data*, *saved_caches*, and *hints* directories
>>    of these nodes.
>>    3. Make the first node to add into the *dcNew* a seed node. Set the
>>    seed list of the first node with its IP address and the IP addresses of the
>>    other seed nodes in the cluster.
>>    4. Set the *initial_token* setting for the first node. You can
>>    calculate the values using the algorithm in my blog post:
>>    https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html.
>>    For convenience I have calculated them:
>>    *-9223372036854775808,-4611686018427387904,0,4611686018427387904*.
>>    Note, remove the *allocate_tokens_for_keyspace* setting from the
>>    *cassandra.yaml* file for this (seed) node.
>>    5. Check to make sure that no other node in the cluster is assigned
>>    any of the four tokens specified above. If there is another node in the
>>    cluster that is assigned one of the above tokens, increment the conflicting
>>    token by values of one until no other node in the cluster is assigned that
>>    token value. The idea is to make sure that these four tokens are unique to
>>    the node.
>>    6. Add the seed node to cluster. Make sure it is listed in *dcNew *by
>>    checking nodetool status.
>>    7. Create a dummy keyspace in *dcNew* that has a replication factor
>>    of 2.
>>    8. Set the *allocate_tokens_for_keyspace* value to be the name of the
>>    dummy keyspace for the other two nodes you want to add to *dcNew*.
>>    Note remove the *initial_token* setting for these other nodes.
>>    9. Set *auto_bootstrap* to *false* for the other two nodes you want
>>    to add to *dcNew*.
>>    10. Add the other two nodes to the cluster, one at a time.
>>    11. If you are happy with the distribution, copy the data to *dcNew*
>>    by running a rebuild.
>>
>>
>> Hope this helps.
>>
>> Regards,
>> Anthony
>>
>> On Fri, 29 Nov 2019 at 02:08, Enrico Cavallin <cavallin.enrico@gmail.com>
>> wrote:
>>
>>> Hi all,
>>> I have an old datacenter with 4 nodes and 256 tokens each.
>>> I am now starting a new datacenter with 3 nodes and num_token=4
>>> and allocate_tokens_for_keyspace=myBiggestKeyspace in each node.
>>> Both DCs run Cassandra 3.11.x.
>>>
>>> myBiggestKeyspace has RF=3 in dcOld and RF=2 in dcNew. Now dcNew is very
>>> unbalanced.
>>> Also keyspaces with RF=2 in both DCs have the same problem.
>>> Did I miss something or even with  allocate_tokens_for_keyspace I have
>>> strong limitations with low num_token?
>>> Any suggestions on how to mitigate it?
>>>
>>> # nodetool status myBiggestKeyspace
>>> Datacenter: dcOld
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address   Load       Tokens       Owns (effective)  Host ID
>>>                       Rack
>>> UN  x.x.x.x  515.83 GiB  256          76.2%
>>> fc462eb2-752f-4d26-aae3-84cb9c977b8a  rack1
>>> UN  x.x.x.x  504.09 GiB  256          72.7%
>>> d7af8685-ba95-4854-a220-bc52dc242e9c  rack1
>>> UN  x.x.x.x  507.50 GiB  256          74.6%
>>> b3a4d3d1-e87d-468b-a7d9-3c104e219536  rack1
>>> UN  x.x.x.x  490.81 GiB  256          76.5%
>>> 41e80c5b-e4e3-46f6-a16f-c784c0132dbc  rack1
>>>
>>> Datacenter: dcNew
>>> ==============
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  Address    Load       Tokens       Owns (effective)  Host ID
>>>                        Rack
>>> UN  x.x.x.x   145.47 KiB  4            56.3%
>>> 7d089351-077f-4c36-a2f5-007682f9c215  rack1
>>> UN  x.x.x.x   122.51 KiB  4            55.5%
>>> 625dafcb-0822-4c8b-8551-5350c528907a  rack1
>>> UN  x.x.x.x   127.53 KiB  4            88.2%
>>> c64c0ce4-2f85-4323-b0ba-71d70b8e6fbf  rack1
>>>
>>> Thanks,
>>> -- ec
>>>
>>

Mime
View raw message