cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Fontana Oscarsson <simon.fontana.oscars...@ericsson.com>
Subject Re: Best approach to prepare to shutdown a cassandra node
Date Mon, 16 Oct 2017 11:54:08 GMT
Looking at the code in trunk, the stopdemon command invokes the 
CassandraDaemon.stop() function which does a graceful shutdown by 
stopping jmxServer and drains the node by the shutdown hook.

/Simon

On 2017-10-13 20:42, Javier Canillas wrote:
> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>
> Or did it change?
>
> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee <anshu.vajpayee@gmail.com 
> <mailto:anshu.vajpayee@gmail.com>>:
>
>     Why are you killing when we have nodetool stopdaemon ?
>
>     On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
>     <javier.canillas@gmail.com <mailto:javier.canillas@gmail.com>> wrote:
>
>         That's what I thought.
>
>         Thanks!
>
>         2017-10-12 14:26 GMT-03:00 Hannu Kröger <hkroger@gmail.com
>         <mailto:hkroger@gmail.com>>:
>
>             Hi,
>
>             Drain should be enough.  It stops accepting writes and
>             after that cassandra can be safely shut down.
>
>             Hannu
>
>             On 12 October 2017 at 20:24:41, Javier Canillas
>             (javier.canillas@gmail.com
>             <mailto:javier.canillas@gmail.com>) wrote:
>
>>             Hello everyone,
>>
>>             I have some time working with Cassandra, but every time I
>>             need to shutdown a node (for any reason like upgrading
>>             version or moving instance to another host) I see several
>>             errors on the client applications (yes, I'm using the
>>             official java driver).
>>
>>             By the way, I'm starting C* as a stand-alone process
>>             <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>             and C* version is 3.11.0.
>>
>>             The way I have implemented the shutdown process is
>>             something like the following:
>>
>>             /# Drain all information from commitlog into sstables/
>>             /bin/nodetool drain
>>             /
>>             /
>>             /
>>             /cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep
>>             -v "grep"|awk '{print $2}'`
>>             /
>>             /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne
>>             "1" ]; then/
>>             /        echo "Asking Cassandra to shutdown (nodetool
>>             drain doesn't stop cassandra)"/
>>             /        kill $cassandra_pid/
>>             /
>>             /
>>             /        echo -n "+ Checking it is down. "/
>>             /counter=10/
>>             /        while [ "$counter" -ne 0 -a ! kill -0
>>             $cassandra_pid > /dev/null 2>&1 ]/
>>             /        do/
>>             /echo -n ". "/
>>             /((counter--))/
>>             /sleep 1s/
>>             /        done/
>>             /        echo ""/
>>             /        if ! kill -0 $cassandra_pid > /dev/null 2>&1;
then/
>>             /echo "+ Its down."/
>>             /        else/
>>             /echo "- Killing Cassandra."/
>>             /kill -9 $cassandra_pid/
>>             /        fi/
>>             /else/
>>             /        echo "Care there was a problem finding Cassandra
>>             PID"/
>>             /fi/
>>             /
>>             /
>>             Should I add at the beginning the following lines?
>>
>>             echo "shutdowing cassandra gracefully with: nodetool
>>             disable gossip"
>>             $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>>             echo "shutdowing cassandra gracefully with: nodetool
>>             disable binary protocol"
>>             $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>>             echo "shutdowing cassandra gracefully with: nodetool thrift"
>>             $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>
>>             The shutdown log is the following:
>>
>>             /WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>             14:20:52,343 StorageService.java:321 - Stopping gossip by
>>             operator request/
>>             /INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>             14:20:52,344 Gossiper.java:1532 - Announcing shutdown/
>>             /INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>             14:20:52,355 StorageService.java:2268 - Node
>>             /10.254.169.36 <http://10.254.169.36> state jump to shutdown/
>>             /INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
>>             14:20:56,141 Server.java:176 - Stop listening for CQL
>>             clients/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:20:59,472 StorageService.java:1442 - DRAINING:
>>             starting drain process/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:20:59,474 HintsService.java:220 - Paused hints dispatch/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:20:59,477 Gossiper.java:1532 - Announcing shutdown/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:20:59,480 StorageService.java:2268 - Node /127.0.0.1
>>             <http://127.0.0.1> state jump to shutdown/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:21:01,483 MessagingService.java:984 - Waiting for
>>             messaging service to quiesce/
>>             /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
>>             2017-10-12 14:21:01,485 MessagingService.java:1338 -
>>             MessagingService has terminated the accept() thread/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:21:02,095 HintsService.java:220 - Paused hints dispatch/
>>             /INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>             14:21:02,111 StorageService.java:1442 - DRAINED/
>>
>>             Disabling Gossip seemed a good idea, but watching the
>>             logs, it may use it to gracefully telling the other nodes
>>             he is going down, so I don't know if it's good or bad idea.
>>
>>             Disabling Thrift and Binary protocol should only avoid
>>             new connections, but the one stablished and running
>>             should be attempted to finish.
>>
>>             Any thoughts or comments?
>>
>>             Thanks
>>
>>             Javier.
>>
>>
>
>
>
>
>     -- 
>     *Regards,*
>     *Anshu *
>     *
>     *
>
>


Mime
View raw message