Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 70D5910E88 for ; Fri, 5 Dec 2014 16:20:13 +0000 (UTC) Received: (qmail 96968 invoked by uid 500); 5 Dec 2014 16:20:13 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 96931 invoked by uid 500); 5 Dec 2014 16:20:13 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 96919 invoked by uid 99); 5 Dec 2014 16:20:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Dec 2014 16:20:13 +0000 Date: Fri, 5 Dec 2014 16:20:13 +0000 (UTC) From: "Philip Thompson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-8428) Nodetool Drain kills C* Process MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235683#comment-14235683 ] Philip Thompson commented on CASSANDRA-8428: -------------------------------------------- {code} INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0 11 StorageService.java:1025 - DRAINING: starting drain process INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0 12 ThriftServer.java:142 - Stop listening to thrift clients INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0 18 Server.java:213 - Stop listening for CQL clients INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0 19 Gossiper.java:1318 - Announcing shutdown INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0 20 MessagingService.java:705 - Waiting for messaging service to quiesce INFO [ACCEPT-localhost/127.0.0.1] 2014-12-05 11:16:56,021 Mess agingService.java:953 - MessagingService has terminated the acc ept() thread INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0 22 ColumnFamilyStore.java:840 - Enqueuing flush of peers: 152 ( 0%) on-heap, 0 (0%) off-heap INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0 23 ColumnFamilyStore.java:840 - Enqueuing flush of compaction_h istory: 1187 (0%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:2] 2014-12-05 11:16:56,023 Memtable. java:325 - Writing Memtable-peers@848262695(0 serialized bytes, 1 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:1] 2014-12-05 11:16:56,024 Memtable. java:325 - Writing Memtable-compaction_history@1754666079(229 s erialized bytes, 9 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:2] 2014-12-05 11:16:56,030 Memtable. java:364 - Completed flushing /Users/philipthompson/cstar/cassa ndra/bin/../data/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-1-Data.db (30 bytes) for commitlog position ReplayPosition(segmentId=1417796200082, position=108607) INFO [MemtableFlushWriter:1] 2014-12-05 11:16:56,031 Memtable.java:364 - Completed flushing /Users/philipthompson/cstar/cassandra/bin/../data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-1-Data.db (239 bytes) for commitlog position ReplayPosition(segmentId=1417796200082, position=108607) ERROR [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,087 StorageService.java:383 - Stopping gossiper WARN [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,087 StorageService.java:291 - Stopping gossip by operator request INFO [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,088 Gossiper.java:1318 - Announcing shutdown ERROR [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:58,096 CommitLog.java:367 - Failed managing commit log segments. Commit disk failure policy is stop; terminating thread org.apache.cassandra.io.FSWriteError: java.nio.channels.ClosedByInterruptException at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:177) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:365) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$3.call(CommitLogSegmentManager.java:369) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$3.call(CommitLogSegmentManager.java:366) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:153) ~[main/:na] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [main/:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] Caused by: java.nio.channels.ClosedByInterruptException: null at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[na:1.7.0_67] at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:919) ~[na:1.7.0_67] at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:166) ~[main/:na] ... 6 common frames omitted INFO [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:58,096 StorageService.java:1025 - DRAINED {code} Is what I see in 2.1.2 > Nodetool Drain kills C* Process > ------------------------------- > > Key: CASSANDRA-8428 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8428 > Project: Cassandra > Issue Type: Bug > Reporter: Philip Thompson > Assignee: Brandon Williams > Labels: nodetool > Fix For: 2.0.12 > > Attachments: system.log > > > Nodetool Drain is documented at http://wiki.apache.org/cassandra/NodeTool and in the nodetool help to flush a node and stop accepting writes. This is the behavior I see with 2.1.2. > In 2.0.11 and 1.2.19, instead the Cassandra Process is killed. In the 1.2.19 logs, I see: > {code} > INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,234 StorageService.java (line 964) > DRAINING: starting drain process > INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,235 ThriftServer.java (line 116) S > top listening to thrift clients > INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,239 Server.java (line 159) Stop li > stening for CQL clients > INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,239 Gossiper.java (line 1203) Announcing shutdown > INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:46,240 MessagingService.java (line 696) Waiting for messaging service to quiesce > INFO [ACCEPT-/127.0.0.1] 2014-12-05 10:32:46,241 MessagingService.java (line 919) MessagingService shutting down server thread.{code} > So it appears this in an intentional shut down, in which case the docs and help are wrong. I could not find a JIRA that described the change in behavior moving to 2.1. > Other users on IRC report that drain works as expected for them on 1.2.19. Attached are 2.0 logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)