incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: Nodetool drain automatically shutting down node?
Date Mon, 11 Mar 2013 13:34:59 GMT
+1, we ran into the same issue.  From docs, nodetool drain shuts down ports which then prevent
nodetool from working as it needs to talk to cassandra through the ports.

I asked this same question 2 weeks ago but didn't get an answer.  I am guessing but I think
1.2.2 seems to fix this a little better so it doesn't complete shut down(ie. I think in 1.2.2
you can still use nodetool after a drain though I need to retest this).

A workaround for this issue though is enable your iptables to isolate all nodes and allow
now communication in or out on ports 9160 and 7199, do a drain, start cassandra back up while
it is isolated and then snapshot the cluster.

OR

Allow 1.2.2 to read all the commit logs which is what we did (we did snapshot then drain and
created hardlinks to the commit log files as well)

Take your pick.  I have a feeling they are trying to make this better in the newer releases
so we can drain and then snapshot though I would have been fine with a snapshot and no need
to drain I think as in our testing the drain did NOT drain.  It did move everything in the
commit log to sstables but the commit log files where still all there and where the same size
AND they all got replayed on startup again.  I highly suggest to QA all this yourself and
us "du –sh /…/…/commitlog" and "du –sh /…/…/data_files" so you get a feel on what
you should expect.

Dean

From: Andrew Bialecki <andrew.bialecki@gmail.com<mailto:andrew.bialecki@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Friday, March 8, 2013 8:36 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Nodetool drain automatically shutting down node?

Hey all,

We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're testing the upgrade
process on our dev cluster. We turned off all client access to the cluster and then ran "nodetool
drain" on the first instance with the intention of running "nodetool snapshot" once it finished.
However, after running the drain, didn't see any errors, but the Cassandra process was no
longer running. Is that expected? From everything I've read it doesn't seem like it, but maybe
I'm mistaken.

Here's the relevant portion of the log from that node (notice it says it's shutting down the
server thread in there):

INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 StorageService.java (line
774) DRAINING: starting drain process
 INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 CassandraDaemon.java
(line 218) Stop listening to thrift clients
 INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315 Gossiper.java (line
1133) Announcing shutdown
 INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318 MessagingService.java
(line 534) Waiting for messaging service to quiesce
 INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143<http://10.116.111.143>]
2013-03-09 03:26:49,319 MessagingService.java (line 690) MessagingService shutting down server
thread.
 INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-Counter1@177255852(14810190/60139556 serialized/live
bytes, 243550 ops)
 INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264) Writing Memtable-Counter1@177255852(14810190/60139556
serialized/live bytes, 243550 ops)
 INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305) Completed flushing
/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db (15204741 bytes)
for commitlog position ReplayPosition(segmentId=1362797442799, position=27621115)
 INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java (line 109) Compacting
[SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')]
 INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193 StorageService.java
(line 774) DRAINED

Thanks in advanced for any help.

Cheers,
Andrew

Mime
View raw message