cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Briggs <james.bri...@yahoo.com>
Subject Re: Blocking while a node finishes joining the cluster after restart.
Date Tue, 16 Sep 2014 19:21:45 GMT
FYI: OpsCenter has a default of sleep 60 seconds after each node restart,
and an option of "drain before stopping."


I haven't noticed if they do anything special with seeds.
(At least one seed needs to be running before you restart other nodes.)


I wondered the same thing as Kevin and came to these conclusions.

Fixing the startup script is non-trivial as far as startup scripts go.

For start, it would have to:

- parse cassandra.yaml for seeds
- if itself is not a seed, wait for a seed to start first. (could take minutes or never.)

- continue start.


For a no-downtime cluster restart script, it would have to:

- verify cluster health (ie. quorum/CL is met or you lose writes)

- parse cassandra.yaml for seeds and see if a seed is up
- stop gossip and thrift
- maybe do compaction before drain

- drain node
- stop/start or restart cassandra process.

http://comments.gmane.org/gmane.comp.db.cassandra.user/20144

Both of those scripts would be nice to have. :)

OpsCenter is flaky at doing rolling restart in my test cluster,
so an alternative is needed.

Also, the free OpsCenter doesn't have rolling repair option enabled.

ccm has the options to do drain, stop and start, but a bash
script would be needed to make it rolling.

https://github.com/pcmanus/ccm


Thanks, James. 
-- 
Cassandra/MySQL DBA. Available in San Jose area or remote.



________________________________
 From: Duncan Sands <duncan.sands@gmail.com>
To: user@cassandra.apache.org 
Sent: Tuesday, September 16, 2014 11:09 AM
Subject: Re: Blocking while a node finishes joining the cluster after restart.
 

Hi Kevin, if you are using the latest version of opscenter, then even the 
community (= free) edition can do a rolling restart of your cluster.  It's 
pretty convenient.

Ciao, Duncan.

On 16/09/14 19:44, Kevin Burton wrote:
> Say I want to do a rolling restart of Cassandra…
>
> I can’t just restart all of them because they need some time to gossip and for
> that gossip to get to all nodes.
>
> What is the best strategy for this.
>
> It would be something like:
>
> /etc/init.d/cassandra restart && wait-for-cassandra.sh
>
> … or something along those lines.
>
> --
>
> Founder/CEO Spinn3r.com <http://Spinn3r.com>

> Location: *San Francisco, CA*
> blog:**http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com
>
>
Mime
View raw message