cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Grasso <anthony.gra...@gmail.com>
Subject Re: [EXTERNAL] How to reduce vnodes without downtime
Date Mon, 03 Feb 2020 02:35:40 GMT
Hi Sergio,

There is a misunderstanding here. My post makes no recommendation for the
value of num_tokens. Rather, it focuses on how to use
the allocate_tokens_for_keyspace setting when creating a new cluster.

Whilst a value of 4 is used for num_tokens in the post, it was chosen for
demonstration purposes. Specifically it makes:

   - the uneven token distribution in a small cluster very obvious,
   - identifying the endpoints displayed in nodetool ring easy, and
   - the initial_token setup less verbose and easier to follow.

I will add an editorial note to the post with the above information
so there is no confusion about why 4 tokens were used.

I would only consider moving a cluster to 4 tokens if it is larger than 100
nodes. If you read through the paper that Erick mentioned, written by Joe
Lynch & Josh Snyder, they show that the num_tokens impacts the availability
of large scale clusters.

If you are after more details about the trade-offs between different sized
token values, please see the discussion on the dev mailing list: "[Discuss]
num_tokens default in Cassandra 4.0
<https://www.mail-archive.com/search?l=dev%40cassandra.apache.org&q=subject%3A%22%5C%5BDiscuss%5C%5D+num_tokens+default+in+Cassandra+4.0%22&o=oldest>
".

Regards,
Anthony

On Sat, 1 Feb 2020 at 10:07, Sergio <lapostadisergio@gmail.com> wrote:

>
> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
This
> is the article with 4 token recommendations.
> @Erick Ramirez. which is the dev thread for the default 32 tokens
> recommendation?
>
> Thanks,
> Sergio
>
> Il giorno ven 31 gen 2020 alle ore 14:49 Erick Ramirez <
> flightctlr@gmail.com> ha scritto:
>
>> There's an active discussion going on right now in a separate dev thread.
>> The current "default recommendation" is 32 tokens. But there's a push for 4
>> in combination with allocate_tokens_for_keyspace from Jon Haddad & co
>> (based on a paper from Joe Lynch & Josh Snyder).
>>
>> If you're satisfied with the results from your own testing, go with 4
>> tokens. And that's the key -- you must test, test, TEST! Cheers!
>>
>> On Sat, Feb 1, 2020 at 5:17 AM Arvinder Dhillon <dhillonarvi@gmail.com>
>> wrote:
>>
>>> What is recommended vnodes now? I read 8 in later cassandra 3.x
>>> Is the new recommendation 4 now even in version 3.x (asking for 3.11)?
>>> Thanks
>>>
>>> On Fri, Jan 31, 2020 at 9:49 AM Durity, Sean R <
>>> SEAN_R_DURITY@homedepot.com> wrote:
>>>
>>>> These are good clarifications and expansions.
>>>>
>>>>
>>>>
>>>> Sean Durity
>>>>
>>>>
>>>>
>>>> *From:* Anthony Grasso <anthony.grasso@gmail.com>
>>>> *Sent:* Thursday, January 30, 2020 7:25 PM
>>>> *To:* user <user@cassandra.apache.org>
>>>> *Subject:* Re: [EXTERNAL] How to reduce vnodes without downtime
>>>>
>>>>
>>>>
>>>> Hi Maxim,
>>>>
>>>>
>>>>
>>>> Basically what Sean suggested is the way to do this without downtime.
>>>>
>>>>
>>>>
>>>> To clarify the, the *three* steps following the "Decommission each
>>>> node in the DC you are working on" step should be applied to *only*
>>>> the decommissioned nodes. So where it say "*all nodes*" or "*every
>>>> node*" it applies to only the decommissioned nodes.
>>>>
>>>>
>>>>
>>>> In addition, the step that says "Wipe data on all the nodes", I would
>>>> delete all files in the following directories on the decommissioned nodes.
>>>>
>>>>    - data (usually located in /var/lib/cassandra/data)
>>>>    - commitlogs (usually located in /var/lib/cassandra/commitlogs)
>>>>    - hints (usually located in /var/lib/casandra/hints)
>>>>    - saved_caches (usually located in /var/lib/cassandra/saved_caches)
>>>>
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Anthony
>>>>
>>>>
>>>>
>>>> On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <
>>>> SEAN_R_DURITY@homedepot.com> wrote:
>>>>
>>>> Your procedure won’t work very well. On the first node, if you switched
>>>> to 4, you would end up with only a tiny fraction of the data (because the
>>>> other nodes would still be at 256). I updated a large cluster (over 150
>>>> nodes – 2 DCs) to smaller number of vnodes. The basic outline was this:
>>>>
>>>>
>>>>
>>>>    - Stop all repairs
>>>>    - Make sure the app is running against one DC only
>>>>    - Change the replication settings on keyspaces to use only 1 DC
>>>>    (basically cutting off the other DC)
>>>>    - Decommission each node in the DC you are working on. Because the
>>>>    replication setting are changed, no streaming occurs. But it releases
the
>>>>    token assignments
>>>>    - Wipe data on all the nodes
>>>>    - Update configuration on every node to your new settings,
>>>>    including auto_bootstrap = false
>>>>    - Start all nodes. They will choose tokens, but not stream any data
>>>>    - Update replication factor for all keyspaces to include the new DC
>>>>    - I disabled binary on those nodes to prevent app connections
>>>>    - Run nodetool reduild with -dc (other DC) on as many nodes as your
>>>>    system can safely handle until they are all rebuilt.
>>>>    - Re-enable binary (and app connections to the rebuilt DC)
>>>>    - Turn on repairs
>>>>    - Rest for a bit, then reverse the process for the remaining DCs
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Sean Durity – Staff Systems Engineer, Cassandra
>>>>
>>>>
>>>>
>>>> *From:* Maxim Parkachov <lazy.gopher@gmail.com>
>>>> *Sent:* Thursday, January 30, 2020 10:05 AM
>>>> *To:* user@cassandra.apache.org
>>>> *Subject:* [EXTERNAL] How to reduce vnodes without downtime
>>>>
>>>>
>>>>
>>>> Hi everyone,
>>>>
>>>>
>>>>
>>>> with discussion about reducing default vnodes in version 4.0 I would
>>>> like to ask, what would be optimal procedure to perform reduction of vnodes
>>>> in existing 3.11.x cluster which was set up with default value 256. Cluster
>>>> has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could
>>>> not add more servers, nor to create additional DC, everything is physical.
>>>> This should be done without downtime.
>>>>
>>>>
>>>>
>>>> My idea for such procedure would be
>>>>
>>>>
>>>>
>>>> for each node:
>>>>
>>>> - decommission node
>>>>
>>>> - set auto_bootstrap to true and vnodes to 4
>>>>
>>>> - start and wait till node joins cluster
>>>>
>>>> - run cleanup on rest of nodes in cluster
>>>>
>>>> - run repair on whole cluster (not sure if needed after cleanup)
>>>>
>>>> - set auto_bootstrap to false
>>>>
>>>> repeat for each node
>>>>
>>>>
>>>>
>>>> rolling restart of cluster
>>>>
>>>> cluster repair
>>>>
>>>>
>>>>
>>>> Is this sounds right ? My concern is that after decommission, node will
>>>> start on the same IP which could create some confusion.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Maxim.
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>>
>>>> The information in this Internet Email is confidential and may be
>>>> legally privileged. It is intended solely for the addressee. Access to this
>>>> Email by anyone else is unauthorized. If you are not the intended
>>>> recipient, any disclosure, copying, distribution or any action taken or
>>>> omitted to be taken in reliance on it, is prohibited and may be unlawful.
>>>> When addressed to our clients any opinions or advice contained in this
>>>> Email are subject to the terms and conditions expressed in any applicable
>>>> governing The Home Depot terms of business or client engagement letter. The
>>>> Home Depot disclaims all responsibility and liability for the accuracy and
>>>> content of this attachment and for any damages or losses arising from any
>>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>>>> items of a destructive nature, which may be contained in this attachment
>>>> and shall not be liable for direct, indirect, consequential or special
>>>> damages in connection with this e-mail message or its attachment.
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> The information in this Internet Email is confidential and may be
>>>> legally privileged. It is intended solely for the addressee. Access to this
>>>> Email by anyone else is unauthorized. If you are not the intended
>>>> recipient, any disclosure, copying, distribution or any action taken or
>>>> omitted to be taken in reliance on it, is prohibited and may be unlawful.
>>>> When addressed to our clients any opinions or advice contained in this
>>>> Email are subject to the terms and conditions expressed in any applicable
>>>> governing The Home Depot terms of business or client engagement letter. The
>>>> Home Depot disclaims all responsibility and liability for the accuracy and
>>>> content of this attachment and for any damages or losses arising from any
>>>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>>>> items of a destructive nature, which may be contained in this attachment
>>>> and shall not be liable for direct, indirect, consequential or special
>>>> damages in connection with this e-mail message or its attachment.
>>>>
>>>

Mime
View raw message