cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13196) test failure in snitch_test.TestGossipingPropertyFileSnitch.test_prefer_local_reconnect_on_listen_address
Date Sun, 12 Mar 2017 22:15:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906715#comment-15906715
] 

Jeff Jirsa commented on CASSANDRA-13196:
----------------------------------------

There's a real risk (in large clusters, or in clusters with large schemas, or when upgrading
versions where we run in a mixed-version state) that we can have a lot of migrationtasks in
flight, so much so that we can actually kill nodes ( see CASSANDRA-11748 for example ) - re-queueing
more migration tasks when one times out is a good way to make the problem worse, not better.
I'm very concerned with the approach [here|https://github.com/Gerrrr/cassandra/commit/463f3fecd9348ea0a4ce6eeeb30141527b8b10eb#diff-f484a759f797776d9cc5d8af92b29e5eR156]
where we just blindly schedule another poll. 

Do we even know why this failed in the first place? Isn't the right fix understanding why
all 3 migration tasks failed, not just making more and more and more migration tasks?

> test failure in snitch_test.TestGossipingPropertyFileSnitch.test_prefer_local_reconnect_on_listen_address
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13196
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13196
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Michael Shuler
>            Assignee: Aleksandr Sorokoumov
>              Labels: dtest, test-failure
>         Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log,
node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1487/testReport/snitch_test/TestGossipingPropertyFileSnitch/test_prefer_local_reconnect_on_listen_address
> {code}
> {novnode}
> Error Message
> Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does not exist"
> -------------------- >> begin captured logging << --------------------
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-k6b0iF
> dtest: DEBUG: Done setting configuration options:
> {   'initial_token': None,
>     'num_tokens': '32',
>     'phi_convict_threshold': 5,
>     'range_request_timeout_in_ms': 10000,
>     'read_request_timeout_in_ms': 10000,
>     'request_timeout_in_ms': 10000,
>     'truncate_request_timeout_in_ms': 10000,
>     'write_request_timeout_in_ms': 10000}
> cassandra.policies: INFO: Using datacenter 'dc1' for DCAwareRoundRobinPolicy (via host
'127.0.0.1'); if incorrect, please specify a local_dc to the constructor, or limit contact
points to local cluster nodes
> cassandra.cluster: INFO: New Cassandra host <Host: 127.0.0.1 dc1> discovered
> --------------------- >> end captured logging << ---------------------
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
>     testMethod()
>   File "/home/automaton/cassandra-dtest/snitch_test.py", line 87, in test_prefer_local_reconnect_on_listen_address
>     new_rows = list(session.execute("SELECT * FROM {}".format(stress_table)))
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 1998, in execute
>     return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile,
paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 3784, in result
>     raise self._final_exception
> 'Error from server: code=2200 [Invalid query] message="keyspace keyspace1 does not exist"\n--------------------
>> begin captured logging << --------------------\ndtest: DEBUG: cluster ccm directory:
/tmp/dtest-k6b0iF\ndtest: DEBUG: Done setting configuration options:\n{   \'initial_token\':
None,\n    \'num_tokens\': \'32\',\n    \'phi_convict_threshold\': 5,\n    \'range_request_timeout_in_ms\':
10000,\n    \'read_request_timeout_in_ms\': 10000,\n    \'request_timeout_in_ms\': 10000,\n
   \'truncate_request_timeout_in_ms\': 10000,\n    \'write_request_timeout_in_ms\': 10000}\ncassandra.policies:
INFO: Using datacenter \'dc1\' for DCAwareRoundRobinPolicy (via host \'127.0.0.1\'); if incorrect,
please specify a local_dc to the constructor, or limit contact points to local cluster nodes\ncassandra.cluster:
INFO: New Cassandra host <Host: 127.0.0.1 dc1> discovered\n--------------------- >>
end captured logging << ---------------------'
> {novnode}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message