cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Bestland (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10730) periodic timeout errors in dtest
Date Tue, 08 Dec 2015 18:37:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047187#comment-15047187
] 

Greg Bestland edited comment on CASSANDRA-10730 at 12/8/15 6:36 PM:
--------------------------------------------------------------------

[~aholmber]

Not that specific issue no. We have seem similar timeout issues when the disk on the instances
we were using was extremely slow. I didn't investigate but I doubt it was due to a protocol
issue with the server rather just env or performance related.

I have seen quite a few intermittent non specific timeout issues that manifest themselves
in the following way

{code}
 File "/home/jenkins/.pyenv/versions/2.7.10/lib/python2.7/site-packages/unittest2/case.py",
line 67, in testPartExecutor
    yield
  File "/home/jenkins/.pyenv/versions/2.7.10/lib/python2.7/site-packages/unittest2/case.py",
line 625, in run
    testMethod()
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/tests/integration/standard/test_metadata.py",
line 430, in test_refresh_schema_metadata
    self.session.execute("CREATE TABLE {0}.{1} (a int PRIMARY KEY, b text)".format(self.keyspace_name,
table_name))
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/cassandra/cluster.py",
line 1569, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout).result()
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/cassandra/cluster.py",
line 3122, in result
    raise self._final_exception
"errors={}, last_host=127.0.0.3\n-------------------- >> begin captured logging <<
--------------------\n[DEBUG] 2015-11-25 
{code}

These usually occur around table creates,drops, or keyspace create/dropping. I cannot reproduce
them locally. 




was (Author: greg.bestland):
[~aholmber]

Not that specific issue no. We have seem similar timeout issues when the disk on the instances
we were using was extremely slow.

I have seen quite a few intermittent non specific timeout issues that manifest themselves
in the following way.

{code}
 File "/home/jenkins/.pyenv/versions/2.7.10/lib/python2.7/site-packages/unittest2/case.py",
line 67, in testPartExecutor
    yield
  File "/home/jenkins/.pyenv/versions/2.7.10/lib/python2.7/site-packages/unittest2/case.py",
line 625, in run
    testMethod()
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/tests/integration/standard/test_metadata.py",
line 430, in test_refresh_schema_metadata
    self.session.execute("CREATE TABLE {0}.{1} (a int PRIMARY KEY, b text)".format(self.keyspace_name,
table_name))
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/cassandra/cluster.py",
line 1569, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout).result()
  File "/home/jenkins/workspace/datastax.python-driver.master/f50f1487/cassandra/cluster.py",
line 3122, in result
    raise self._final_exception
"errors={}, last_host=127.0.0.3\n-------------------- >> begin captured logging <<
--------------------\n[DEBUG] 2015-11-25 
{code}

These usually occur around table creates,drops, or keyspace create/dropping. I cannot reproduce
them locally. 



> periodic timeout errors in dtest
> --------------------------------
>
>                 Key: CASSANDRA-10730
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10730
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jim Witschey
>            Assignee: Jim Witschey
>
> Dtests often fail with connection timeout errors. For example:
> http://cassci.datastax.com/job/cassandra-3.1_dtest/lastCompletedBuild/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3/deletion_test/
> {code}
> ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=Timed out
creating connection (10 seconds), last_host=None',)})
> {code}
> We've merged a PR to increase timeouts:
> https://github.com/riptano/cassandra-dtest/pull/663
> It doesn't look like this has improved things:
> http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/363/testReport/
> Next steps here are
> * to scrape Jenkins history to see if and how the number of tests failing this way has
increased (it feels like it has). From there we can bisect over the dtests, ccm, or C*, depending
on what looks like the source of the problem.
> * to better instrument the dtest/ccm/C* startup process to see why the nodes start but
don't successfully make the CQL port available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message