incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Nodetool drain automatically shutting down node?
Date Mon, 11 Mar 2013 13:52:13 GMT
Drain stops listening for connections from client and other nodes, and flushes all the data
to disk. The purpose is to get everything into SSTables, so we do not want to process any
more writes. 

The error is logged at DEBUG as it's not important, just means a thread (the processed gossip)
was cancelled. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 8/03/2013, at 7:58 PM, Andrew Bialecki <andrew.bialecki@gmail.com> wrote:

> If it's helps, here's the log with debug log statements. Possibly issue with that exception?
> 
> INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,402 StorageService.java
(line 774) DRAINING: starting drain process
>  INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,403 CassandraDaemon.java
(line 218) Stop listening to thrift clients
>  INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,404 Gossiper.java (line
1133) Announcing shutdown
> DEBUG [GossipTasks:1] 2013-03-09 03:54:33,328 DebuggableThreadPoolExecutor.java (line
190) Task cancelled
> java.util.concurrent.CancellationException
> 	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> 	at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:182)
> 	at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:146)
> 	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 StorageService.java
(line 776) DRAINING: shutting down MessageService
>  INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406 MessagingService.java
(line 534) Waiting for messaging service to quiesce
>  INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 03:54:33,407
MessagingService.java (line 690) MessagingService shutting down server thread.
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 StorageService.java
(line 776) DRAINING: waiting for streaming
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408 StorageService.java
(line 776) DRAINING: clearing mutation stage
> DEBUG [Thread-5] 2013-03-09 03:54:33,408 Gossiper.java (line 221) Reseting version for
/10.83.55.44
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 StorageService.java
(line 776) DRAINING: flushing column families
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409 ColumnFamilyStore.java
(line 713) forceFlush requested but everything is clean in Counter1
> DEBUG [Thread-6] 2013-03-09 03:54:33,410 Gossiper.java (line 221) Reseting version for
/10.80.187.124
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java
(line 713) forceFlush requested but everything is clean in Super1
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java
(line 713) forceFlush requested but everything is clean in SuperCounter1
> DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410 ColumnFamilyStore.java
(line 713) forceFlush requested but everything is clean in Standard1
>  INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,510 StorageService.java
(line 774) DRAINED
> 
> On Fri, Mar 8, 2013 at 10:36 PM, Andrew Bialecki <andrew.bialecki@gmail.com> wrote:
> Hey all,
> 
> We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're testing the
upgrade process on our dev cluster. We turned off all client access to the cluster and then
ran "nodetool drain" on the first instance with the intention of running "nodetool snapshot"
once it finished. However, after running the drain, didn't see any errors, but the Cassandra
process was no longer running. Is that expected? From everything I've read it doesn't seem
like it, but maybe I'm mistaken.
> 
> Here's the relevant portion of the log from that node (notice it says it's shutting down
the server thread in there):
> 
> INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 StorageService.java
(line 774) DRAINING: starting drain process
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288 CassandraDaemon.java
(line 218) Stop listening to thrift clients
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315 Gossiper.java (line
1133) Announcing shutdown
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318 MessagingService.java
(line 534) Waiting for messaging service to quiesce
>  INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09 03:26:49,319
MessagingService.java (line 690) MessagingService shutting down server thread.
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338 ColumnFamilyStore.java
(line 659) Enqueuing flush of Memtable-Counter1@177255852(14810190/60139556 serialized/live
bytes, 243550 ops)
>  INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264) Writing Memtable-Counter1@177255852(14810190/60139556
serialized/live bytes, 243550 ops)
>  INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305) Completed flushing
/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db (15204741 bytes)
for commitlog position ReplayPosition(segmentId=1362797442799, position=27621115)
>  INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java (line 109)
Compacting [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')]
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193 StorageService.java
(line 774) DRAINED
> 
> Thanks in advanced for any help.
> 
> Cheers,
> Andrew
> 


Mime
View raw message