cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ballet <jbal...@edgelab.ch>
Subject Re: How seed nodes are working and how to upgrade/replace them?
Date Wed, 09 Jan 2019 13:45:24 GMT
On Tue, 8 Jan 2019 at 18:39, Jeff Jirsa <jjirsa@gmail.com> wrote:

> On Tue, Jan 8, 2019 at 8:19 AM Jonathan Ballet <jballet@edgelab.ch> wrote:
>
>> Hi Jeff,
>>
>> thanks for answering to most of my points!
>> From the reloadseeds' ticket, I followed to
>> https://issues.apache.org/jira/browse/CASSANDRA-3829 which was very
>> instructive, although a bit old.
>>
>>
>> On Mon, 7 Jan 2019 at 17:23, Jeff Jirsa <jjirsa@gmail.com> wrote:
>>
>>> > On Jan 7, 2019, at 6:37 AM, Jonathan Ballet <jballet@edgelab.ch>
>>> wrote:
>>> >
>>> [...]
>>>
>>> >   In essence, in my example that would be:
>>> >
>>> >   - decide that #2 and #3 will be the new seed nodes
>>> >   - update all the configuration files of all the nodes to write the
>>> IP addresses of #2 and #3
>>> >   - DON'T restart any node - the new seed configuration will be picked
>>> up only if the Cassandra process restarts
>>> >
>>> > * If I can manage to sort my Cassandra nodes by their age, could it be
>>> a strategy to have the seeds set to the 2 oldest nodes in the cluster?
>>> (This implies these nodes would change as the cluster's nodes get
>>> upgraded/replaced).
>>>
>>> You could do this, seems like a lot of headache for little benefit.
>>> Could be done with simple seed provider and config management
>>> (puppet/chef/ansible) laying  down new yaml or with your own seed provider
>>>
>>
>> So, just to make it clear: sorting by age isn't a goal in itself, it was
>> just an example on how I could get a stable list.
>>
>> Right now, we have a dedicated group of seed nodes + a dedicated group
>> for non-seeds: doing rolling-upgrade of the nodes from the second list is
>> relatively painless (although slow) whereas we are facing the issues
>> discussed in CASSANDRA-3829 for the first group which are non-seeds nodes
>> are not bootstrapping automatically and we need to operate them in a more
>> careful way.
>>
> Rolling upgrade shouldn't need to re-bootstrap. Only replacing a host
> should need a new bootstrap. That should be a new host in your list, so it
> seems like this should be fairly rare?
>

Sorry, that's internal pigdin, by "rolling upgrade" I meant replacing in a
rolling fashion all the nodes.


> What I'm really looking for is a way to simplify adding and removing nodes
>> into our (small) cluster: I can easily provide a small list of nodes from
>> our cluster with our config management tool so that new nodes are
>> discovering the rest of the cluster, but the documentation seems to imply
>> that seed nodes also have other functions and I'm not sure what problems we
>> could face trying to simplify this approach.
>>
>> Ideally, what I would like to have would be:
>>
>> * Considering a stable cluster (no new nodes, no nodes leaving), the N
>> seeds should be always the same N nodes
>> * Adding new nodes should not change that list
>> * Stopping/removing one of these N nodes should "promote" another
>> (non-seed) node as a seed
>>   - that would not restart the already running Cassandra nodes but would
>> update their configuration files.
>>   - if a node restart for whatever reason it would pick up this new
>> configuration
>>
>> So: no node would start its life as a seed, only a few already existing
>> node would have this status. We would not have to deal with the "a seed
>> node doesn't bootstrap" problem and it would make our operation process
>> simpler.
>>
>>
>>> > I also have some more general questions about seed nodes and how they
>>> work:
>>> >
>>> > * I understand that seed nodes are used when a node starts and needs
>>> to discover the rest of the cluster's nodes. Once the node has joined and
>>> the cluster is stable, are seed nodes still playing a role in day to day
>>> operations?
>>>
>>> They’re used probabilistically in gossip to encourage convergence.
>>> Mostly useful in large clusters.
>>>
>>
>> How "large" are we speaking here? How many nodes would it start to be
>> considered "large"?
>>
>
> ~800-1000
>

Alllrriigght, we still have a long way :)

 Jonathan

Mime
View raw message