cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Haddad <jonathan.had...@gmail.com>
Subject Re: Blocking while a node finishes joining the cluster after restart.
Date Fri, 19 Sep 2014 20:05:23 GMT
Depending on how you query (one or quorum) you might be able to do 1 rack at a time (or az
or whatever you've got) assuming your snitch is set up right 


> On Sep 19, 2014, at 11:30 AM, Kevin Burton <burton@spinn3r.com> wrote:
> 
> This is great feedback…
> 
> I think it could actually be even easier than this…
> 
> You could have an ansible (or whatever cluster management system you’re using) role
for just seeds.
> 
> Then you would serially restart all seeds one at a time.  You would need to run ‘nodetool
status’ and make sure the node is ‘U’ (up) I think.. but you might want to make sure
the majority of other nodes have agreed that this node is up and available.
> 
> I think you can ONLY do this serially.. .for a LARGE number of hosts, this might take
a while unless you can compute nodes which have mutually exclusive key ranges.
> 
> The serial approach would take a LONG time for large clusters.  If you have sixty nodes,
it could take an hour to do a rolling restart.
> 
> Kevin
> 
>> On Tue, Sep 16, 2014 at 12:21 PM, James Briggs <james.briggs@yahoo.com> wrote:
>> FYI: OpsCenter has a default of sleep 60 seconds after each node restart,
>> and an option of "drain before stopping."
>> 
>> I haven't noticed if they do anything special with seeds.
>> (At least one seed needs to be running before you restart other nodes.)
>> 
>> I wondered the same thing as Kevin and came to these conclusions.
>> 
>> Fixing the startup script is non-trivial as far as startup scripts go.
>> 
>> For start, it would have to:
>> 
>> - parse cassandra.yaml for seeds
>> - if itself is not a seed, wait for a seed to start first. (could take minutes or
never.)
>> - continue start.
>> 
>> For a no-downtime cluster restart script, it would have to:
>> 
>> - verify cluster health (ie. quorum/CL is met or you lose writes)
>> - parse cassandra.yaml for seeds and see if a seed is up
>> - stop gossip and thrift
>> - maybe do compaction before drain
>> - drain node
>> - stop/start or restart cassandra process.
>> 
>> http://comments.gmane.org/gmane.comp.db.cassandra.user/20144
>> 
>> Both of those scripts would be nice to have. :)
>> 
>> OpsCenter is flaky at doing rolling restart in my test cluster,
>> so an alternative is needed.
>> 
>> Also, the free OpsCenter doesn't have rolling repair option enabled.
>> 
>> ccm has the options to do drain, stop and start, but a bash
>> script would be needed to make it rolling.
>> 
>> https://github.com/pcmanus/ccm
>> 
>> Thanks, James. 
>> -- 
>> Cassandra/MySQL DBA. Available in San Jose area or remote.
>> 
>> From: Duncan Sands <duncan.sands@gmail.com>
>> To: user@cassandra.apache.org 
>> Sent: Tuesday, September 16, 2014 11:09 AM
>> Subject: Re: Blocking while a node finishes joining the cluster after restart.
>> 
>> Hi Kevin, if you are using the latest version of opscenter, then even the 
>> community (= free) edition can do a rolling restart of your cluster.  It's 
>> pretty convenient.
>> 
>> Ciao, Duncan.
>> 
>> On 16/09/14 19:44, Kevin Burton wrote:
>> > Say I want to do a rolling restart of Cassandra…
>> >
>> > I can’t just restart all of them because they need some time to gossip and
for
>> > that gossip to get to all nodes.
>> >
>> > What is the best strategy for this.
>> >
>> > It would be something like:
>> >
>> > /etc/init.d/cassandra restart && wait-for-cassandra.sh
>> >
>> > … or something along those lines.
>> >
>> > --
>> >
>> > Founder/CEO Spinn3r.com <http://Spinn3r.com>
>> 
>> > Location: *San Francisco, CA*
>> > blog:**http://burtonator.wordpress.com
>> > … or check out my Google+ profile
>> 
>> > <https://plus.google.com/102718274791889610666/posts>
>> > <http://spinn3r.com
>> >
>> >
> 
> 
> 
> -- 
> Founder/CEO Spinn3r.com
> Location: San Francisco, CA
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 

Mime
View raw message