cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12509) Shutdown process triggered twice during if the node is drained
Date Fri, 30 Sep 2016 04:14:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534961#comment-15534961
] 

Stefania commented on CASSANDRA-12509:
--------------------------------------

Unfortunately the 3.0 patch has broken [test_archive_commitlog_with_active_commitlog|http://cassci.datastax.com/job/ifesdjeen-12509-3.0-dtest/lastCompletedBuild/testReport/snapshot_test/TestArchiveCommitlog/test_archive_commitlog_with_active_commitlog_2/]
in 3.0. It's a known failure, CASSANDRA-11811, but whilst it currently fails rarely, this
patch makes it fail every single time, even locally.

The problem is that an archived commitlog segment from a destroyed cluster is replayed, which
adds the bootstrap tokens of the old cluster, but subsequently new bootstrap tokens are generated
and the node fails to start because it thinks it has twice as many tokens.

I've proposed this [pull request|https://github.com/riptano/cassandra-dtest/pull/1354] to
fix the problem, which prevents the new cluster from generating new tokens, let's see if it
gets accepted.

> Shutdown process triggered twice during if the node is drained
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-12509
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12509
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alex Petrov
>            Assignee: Alex Petrov
>
> If the node is drained, the {{StorageService#drain}} [method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L4212]
is called, which triggers shutdown of mutation stage, messaging service, compaction, batchlog
etc. In the end of this process, the node is moved to {{DRAINED}} status with the process
still running. 
> When JVM is shutdown, the JVM shutdown hooks are ran, which are subscribed during the
server initialisation: {{Runtime.getRuntime().addShutdownHook(drainOnShutdown);}} [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L575-L636].

> I noticed this behaviour while reviewing [CASSANDRA-12461], as if we'd like add custom
pre and post-shutdown hooks, most likely it makes sense to run them once (or user might expect
such behaviour). 
> Is this behaviour correct? Should we run whole shutdown process twice or just once in
"drain" and no-op during JVM shutdown?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message