cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Branimir Lambov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12277) Extend testing infrastructure to handle expected intermittent flaky tests - see ReplicationAwareTokenAllocatorTest.testNewCluster
Date Tue, 26 Jul 2016 08:15:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393406#comment-15393406
] 

Branimir Lambov commented on CASSANDRA-12277:
---------------------------------------------

If you ask me, the main question here is whether we want to have a test suite that adds value,
or one that is perceived pure. I'd always choose the former over the latter, and the fastest
way to get there is to have an infrastructure to deal with a very common problem in a very
common way, similar to how countless other projects do it.

Tests flake, that's a fact of life. We either deal with it smartly as in, identify where you
can allow it, and let it happen, or the way we have been doing it so far: not know when it
is ok, always retry everything (at significant cost) and ignore real problems.

> Extend testing infrastructure to handle expected intermittent flaky tests - see ReplicationAwareTokenAllocatorTest.testNewCluster
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12277
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12277
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Joshua McKenzie
>            Assignee: Branimir Lambov
>            Priority: Minor
>              Labels: test
>
> From an offline discussion:
> bq. The ReplicationAwareTokenAllocatorTest.testNewCluster failure is a flake -- randomness
will sometimes (on the order of 1/100) cause it to fail. Extending the ranges to avoid these
flakes goes too far and makes the test meaningless.
> bq. How about instead of @flaky/@Ignore which currently indicates a test that intermittently
fails but we do not expect it to, we instead use @tries, or @runs, or some annotation that
indicates "run this thing N times, if M pass we're good". This would allow us to keep the
current "we don't care about these test results (in context of green test board) because intermittent
failures are not expected and the test quality needs shoring up" from "we expect this test
to fail sometimes in this particular way."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message