incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bialecki <andrew.biale...@gmail.com>
Subject Re: Nodetool drain automatically shutting down node?
Date Sat, 09 Mar 2013 03:58:54 GMT
If it's helps, here's the log with debug log statements. Possibly issue
with that exception?

INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,402
StorageService.java (line 774) DRAINING: starting drain process
 INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,403
CassandraDaemon.java (line 218) Stop listening to thrift clients
 INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:32,404
Gossiper.java (line 1133) Announcing shutdown
DEBUG [GossipTasks:1] 2013-03-09 03:54:33,328
DebuggableThreadPoolExecutor.java (line 190) Task cancelled
java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:220)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:182)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:146)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:50)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406
StorageService.java (line 776) DRAINING: shutting down MessageService
 INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,406
MessagingService.java (line 534) Waiting for messaging service to quiesce
 INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09
03:54:33,407 MessagingService.java (line 690) MessagingService shutting
down server thread.
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408
StorageService.java (line 776) DRAINING: waiting for streaming
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,408
StorageService.java (line 776) DRAINING: clearing mutation stage
DEBUG [Thread-5] 2013-03-09 03:54:33,408 Gossiper.java (line 221) Reseting
version for /10.83.55.44
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409
StorageService.java (line 776) DRAINING: flushing column families
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,409
ColumnFamilyStore.java (line 713) forceFlush requested but everything is
clean in Counter1
DEBUG [Thread-6] 2013-03-09 03:54:33,410 Gossiper.java (line 221) Reseting
version for /10.80.187.124
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410
ColumnFamilyStore.java (line 713) forceFlush requested but everything is
clean in Super1
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410
ColumnFamilyStore.java (line 713) forceFlush requested but everything is
clean in SuperCounter1
DEBUG [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,410
ColumnFamilyStore.java (line 713) forceFlush requested but everything is
clean in Standard1
 INFO [RMI TCP Connection(2)-10.116.111.143] 2013-03-09 03:54:33,510
StorageService.java (line 774) DRAINED


On Fri, Mar 8, 2013 at 10:36 PM, Andrew Bialecki
<andrew.bialecki@gmail.com>wrote:

> Hey all,
>
> We're getting ready to upgrade our cluster to 1.2.2 from 1.1.5 and we're
> testing the upgrade process on our dev cluster. We turned off all client
> access to the cluster and then ran "nodetool drain" on the first instance
> with the intention of running "nodetool snapshot" once it finished.
> However, after running the drain, didn't see any errors, but the Cassandra
> process was no longer running. Is that expected? From everything I've read
> it doesn't seem like it, but maybe I'm mistaken.
>
> Here's the relevant portion of the log from that node (notice it says it's
> shutting down the server thread in there):
>
> INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288
> StorageService.java (line 774) DRAINING: starting drain process
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,288
> CassandraDaemon.java (line 218) Stop listening to thrift clients
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:48,315
> Gossiper.java (line 1133) Announcing shutdown
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,318
> MessagingService.java (line 534) Waiting for messaging service to quiesce
>  INFO [ACCEPT-ip-10-116-111-143.ec2.internal/10.116.111.143] 2013-03-09
> 03:26:49,319 MessagingService.java (line 690) MessagingService shutting
> down server thread.
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:49,338
> ColumnFamilyStore.java (line 659) Enqueuing flush of
> Memtable-Counter1@177255852(14810190/60139556 serialized/live bytes,
> 243550 ops)
>  INFO [FlushWriter:7] 2013-03-09 03:26:49,338 Memtable.java (line 264)
> Writing Memtable-Counter1@177255852(14810190/60139556 serialized/live
> bytes, 243550 ops)
>  INFO [FlushWriter:7] 2013-03-09 03:26:49,899 Memtable.java (line 305)
> Completed flushing
> /var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db
> (15204741 bytes) for commitlog position
> ReplayPosition(segmentId=1362797442799, position=27621115)
>  INFO [CompactionExecutor:11] 2013-03-09 03:26:49,900 CompactionTask.java
> (line 109) Compacting
> [SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-102-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-103-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-104-Data.db'),
> SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Counter1/Keyspace1-Counter1-he-101-Data.db')]
>  INFO [RMI TCP Connection(38)-10.116.111.143] 2013-03-09 03:26:50,193
> StorageService.java (line 774) DRAINED
>
>
> Thanks in advanced for any help.
>
> Cheers,
> Andrew
>

Mime
View raw message