cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12277) Extend testing infrastructure to handle expected intermittent flaky tests - see ReplicationAwareTokenAllocatorTest.testNewCluster
Date Mon, 25 Jul 2016 07:09:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391446#comment-15391446
] 

Sylvain Lebresne commented on CASSANDRA-12277:
----------------------------------------------

I'll admit that at face value that seems a weird and potentially dangerous thing to do to
me. I can easily see us defaulting to such mechanism every time we get a flaky test and that
hiding genuine problems more often than not. I don't know the details about that specific
{{ReplicationAwareTokenAllocatorTest.testNewCluster}} test, but if a test knowingly fails
a number of the times, this feels to me like the test isn't precise enough and should be improved.

Note that I can buy that exceptionally a test is such that making it pass 100% of the time
is too much work to be worth the trouble but the test is still nice to have, but that should
be exceptional and I'd rather handle that in a case-by-case basis, with very precise comment
as to why this is allowed to fail some of the times, to make it clear it's a special case.
  

> Extend testing infrastructure to handle expected intermittent flaky tests - see ReplicationAwareTokenAllocatorTest.testNewCluster
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12277
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12277
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Joshua McKenzie
>            Assignee: Branimir Lambov
>            Priority: Minor
>              Labels: test
>
> From an offline discussion:
> bq. The ReplicationAwareTokenAllocatorTest.testNewCluster failure is a flake -- randomness
will sometimes (on the order of 1/100) cause it to fail. Extending the ranges to avoid these
flakes goes too far and makes the test meaningless.
> bq. How about instead of @flaky/@Ignore which currently indicates a test that intermittently
fails but we do not expect it to, we instead use @tries, or @runs, or some annotation that
indicates "run this thing N times, if M pass we're good". This would allow us to keep the
current "we don't care about these test results (in context of green test board) because intermittent
failures are not expected and the test quality needs shoring up" from "we expect this test
to fail sometimes in this particular way."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message