cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: How seed nodes are working and how to upgrade/replace them?
Date Mon, 07 Jan 2019 16:33:28 GMT


> On Jan 7, 2019, at 8:23 AM, Jeff Jirsa <jjirsa@gmail.com> wrote:
> 
> 
> 
> 
>> On Jan 7, 2019, at 6:37 AM, Jonathan Ballet <jballet@edgelab.ch> wrote:
>> 
>> Hi,
>> 
>> I'm trying to understand how seed nodes are working, when and how do they play a
part in a Cassandra cluster, and how they should be managed and propagated to other nodes.
>> 
>> I have a cluster of 6 Cassandra nodes (let's call them #1 to #6), on which node #1
and #2 are seeds. All the configuration files of all the Cassandra nodes are currently configured
with:
>> 
>> ```
>> seed_provider:
>> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>  parameters:
>>  - seeds: 'IP #1,IP #2'
>> ```
>> 
>> We are using a service discovery tool (Consul) which automatically registers new
Cassandra nodes with its dedicated health-check and are able to generate new configuration
based on the content of the service discovery status (with Consul-Template).
>> 
>> 
>> I'm working on how we could improve the upgrades of our servers and how to replace
them completely (new instance with a new IP address).
>> What I would like to do is to replace the machines holding our current seeds (#1
and #2 at the moment) in a rolling upgrade fashion, on a regular basis:
>> 
>> * Is it possible to "promote" any non-seed node as a seed node?
>> 
> 
> Yes - generally you can make any node a seed if you want 
> 
>> * Is it possible to "promote" a new seed node without having to restart all the nodes?
> 
> nodetool reloadseeds

This is apparently in 4.0+ https://issues.apache.org/jira/browse/CASSANDRA-14190


> 
> There are a few weird edge cases where seeds are reloaded automatically and we don’t
document how or why (it’s a side effect of an error condition in hosts going up/down, but
it’s generally pretty minor unless your seed provider is broken)
> 
> 
> (Also true that you could write a seed provider that did this automatically)
> 
> 
>>  In essence, in my example that would be:
>> 
>>  - decide that #2 and #3 will be the new seed nodes
>>  - update all the configuration files of all the nodes to write the IP addresses
of #2 and #3
>>  - DON'T restart any node - the new seed configuration will be picked up only if
the Cassandra process restarts
>> 
>> * If I can manage to sort my Cassandra nodes by their age, could it be a strategy
to have the seeds set to the 2 oldest nodes in the cluster? (This implies these nodes would
change as the cluster's nodes get upgraded/replaced).
> 
> You could do this, seems like a lot of headache for little benefit. Could be done with
simple seed provider and config management (puppet/chef/ansible) laying  down new yaml or
with your own seed provider
> 
>> 
>> 
>> I also have some more general questions about seed nodes and how they work:
>> 
>> * I understand that seed nodes are used when a node starts and needs to discover
the rest of the cluster's nodes. Once the node has joined and the cluster is stable, are seed
nodes still playing a role in day to day operations?
> 
> They’re used probabilistically in gossip to encourage convergence. Mostly useful in
large clusters. 
> 
>> 
>> * The documentation says multiple times that not all nodes should be seed nodes,
but I didn't really find any place about the consequences it has to have "too many" seed nodes.
> 
> Decreases effectiveness of probabilistic gossiping with seed for convergence 
> 
>> Also, relatively to the questions I asked above, is there any downsides of having
changing seed nodes in a cluster? (with the exact same, at some point I define #1 and #2 to
be seeds, then later #4 and #5, etc.)
>> 
> 
> No
> 
>> 
>> Thanks for helping me to understand better how seeds are working!
>> 
>> Jonathan
>> 

Mime
View raw message