cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Thompson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8428) Nodetool Drain kills C* Process
Date Fri, 05 Dec 2014 16:20:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14235683#comment-14235683
] 

Philip Thompson commented on CASSANDRA-8428:
--------------------------------------------

{code}
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0
11 StorageService.java:1025 - DRAINING: starting drain process
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0
12 ThriftServer.java:142 - Stop listening to thrift clients
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0
18 Server.java:213 - Stop listening for CQL clients
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:54,0
19 Gossiper.java:1318 - Announcing shutdown
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0
20 MessagingService.java:705 - Waiting for messaging service to
 quiesce
INFO  [ACCEPT-localhost/127.0.0.1] 2014-12-05 11:16:56,021 Mess
agingService.java:953 - MessagingService has terminated the acc
ept() thread
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0
22 ColumnFamilyStore.java:840 - Enqueuing flush of peers: 152 (
0%) on-heap, 0 (0%) off-heap
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:56,0
23 ColumnFamilyStore.java:840 - Enqueuing flush of compaction_h
istory: 1187 (0%) on-heap, 0 (0%) off-heap
INFO  [MemtableFlushWriter:2] 2014-12-05 11:16:56,023 Memtable.
java:325 - Writing Memtable-peers@848262695(0 serialized bytes,
 1 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:1] 2014-12-05 11:16:56,024 Memtable.
java:325 - Writing Memtable-compaction_history@1754666079(229 s
erialized bytes, 9 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:2] 2014-12-05 11:16:56,030 Memtable.
java:364 - Completed flushing /Users/philipthompson/cstar/cassa
ndra/bin/../data/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-1-Data.db
(30 bytes) for commitlog position ReplayPosition(segmentId=1417796200082, position=108607)
INFO  [MemtableFlushWriter:1] 2014-12-05 11:16:56,031 Memtable.java:364 - Completed flushing
/Users/philipthompson/cstar/cassandra/bin/../data/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-1-Data.db
(239 bytes) for commitlog position ReplayPosition(segmentId=1417796200082, position=108607)
ERROR [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,087 StorageService.java:383 - Stopping gossiper
WARN  [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,087 StorageService.java:291 - Stopping gossip
by operator request
INFO  [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:56,088 Gossiper.java:1318 - Announcing shutdown
ERROR [COMMIT-LOG-ALLOCATOR] 2014-12-05 11:16:58,096 CommitLog.java:367 - Failed managing
commit log segments. Commit disk failure policy is stop; terminating thread
org.apache.cassandra.io.FSWriteError: java.nio.channels.ClosedByInterruptException
        at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:177)
~[main/:na]
        at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:365)
~[main/:na]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$3.call(CommitLogSegmentManager.java:369)
~[main/:na]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$3.call(CommitLogSegmentManager.java:366)
~[main/:na]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:153)
~[main/:na]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [main/:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.nio.channels.ClosedByInterruptException: null
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
~[na:1.7.0_67]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:919) ~[na:1.7.0_67]
        at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:166)
~[main/:na]
        ... 6 common frames omitted
INFO  [RMI TCP Connection(2)-10.150.0.64] 2014-12-05 11:16:58,096 StorageService.java:1025
- DRAINED
{code}
Is what I see in 2.1.2

> Nodetool Drain kills C* Process
> -------------------------------
>
>                 Key: CASSANDRA-8428
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8428
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Brandon Williams
>              Labels: nodetool
>             Fix For: 2.0.12
>
>         Attachments: system.log
>
>
> Nodetool Drain is documented at http://wiki.apache.org/cassandra/NodeTool and in the
nodetool help to flush a node and stop accepting writes. This is the behavior I see with 2.1.2.
> In 2.0.11 and 1.2.19, instead the Cassandra Process is killed. In the 1.2.19 logs, I
see:
> {code}
>  INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,234 StorageService.java
(line 964)
>  DRAINING: starting drain process
>  INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,235 ThriftServer.java (line
116) S
> top listening to thrift clients
>  INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,239 Server.java (line 159)
Stop li
> stening for CQL clients
>  INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:44,239 Gossiper.java (line
1203) Announcing shutdown
>  INFO [RMI TCP Connection(2)-192.168.1.5] 2014-12-05 10:32:46,240 MessagingService.java
(line 696) Waiting for messaging service to quiesce
>  INFO [ACCEPT-/127.0.0.1] 2014-12-05 10:32:46,241 MessagingService.java (line 919) MessagingService
shutting down server thread.{code}
> So it appears this in an intentional shut down, in which case the docs and help are wrong.
I could not find a JIRA that described the change in behavior moving to 2.1.
> Other users on IRC report that drain works as expected for them on 1.2.19. Attached are
2.0 logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message