cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12856) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_cannot_restart_with_different_rack
Date Tue, 03 Jan 2017 09:41:58 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15794639#comment-15794639
] 

Stefania commented on CASSANDRA-12856:
--------------------------------------

I'm trying to reproduce the problem one more time [here|https://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/95/].

Other than the exception above, there isn't anything else in the logs. Therefore, it's not
possible to know for sure what caused the problem unless we can reproduce it but, from source
code inspection, this [line|https://github.com/apache/cassandra/blob/cassandra-3.X/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java#L101]
would explain the problem, if  [{{stop()}}|https://github.com/apache/cassandra/blob/cassandra-3.X/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java#L159]
is called before the thrift server thread has executed it. It doesn't look like it's needed
either, since {{serve()}} is only called once. The test that fails, starts and stops the node
immediately, so this is a valid possibility.

I've prepared the patch for the following branches, I've skipped trunk because Thirft was
removed and 3.11 because it is very similar to 3.X:

||2.1||2.2||3.0||3.X||
|[patch|https://github.com/stef1927/cassandra/tree/12856-2.1]|[patch|https://github.com/stef1927/cassandra/tree/12856-2.2]|[patch|https://github.com/stef1927/cassandra/tree/12856-3.0]|[patch|https://github.com/stef1927/cassandra/tree/12856-3.X]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-3.X-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12856-3.X-dtest/]|

The 2.1 patch applies cleanly on all branches, CI is currently running on 2.1 only. I'm not
sure which branches this patch should be committed to: the race is extremely rare but the
consequences are quite bad (a thread looping forever).


> dtest failure in replication_test.SnitchConfigurationUpdateTest.test_cannot_restart_with_different_rack
> -------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12856
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12856
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Sean McCarthy
>            Assignee: Stefania
>              Labels: dtest, test-failure
>         Attachments: node1.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/280/testReport/replication_test/SnitchConfigurationUpdateTest/test_cannot_restart_with_different_rack
> {code}
> Error Message
> Problem stopping node node1
> {code}{code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
>     testMethod()
>   File "/home/automaton/cassandra-dtest/replication_test.py", line 630, in test_cannot_restart_with_different_rack
>     node1.stop(wait_other_notice=True)
>   File "/usr/local/lib/python2.7/dist-packages/ccmlib/node.py", line 727, in stop
>     raise NodeError("Problem stopping node %s" % self.name)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message