maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Flanigan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
Date Wed, 27 Jan 2016 01:51:39 GMT

    [ https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118455#comment-15118455
] 

Sean Flanigan commented on SUREFIRE-1219:
-----------------------------------------

(Wow, I haven't used that [~seanf] account in years. I'd like to delete it, but that email
address is dead and my password isn't accepted.)

[~tibor17], I certainly don't want to wait until all 100 tests have finished to implement
fail-fast, or I wouldn't have submitted this bug!  In fact, it's when you are forced to use
rerun that fail-fast becomes even more important. Otherwise your test times can double or
triple when there is a common root cause breaking every test.

I'm sorry Tibor, but I'm finding your explanations very difficult to understand.

I don't understand what it is about forking and surefire-junit4/surefire-junit47 which prevents
us from incrementing the counter only when a test has failed _all its runs_ (instead of each
time there is a failure).

If we can only get the right behaviour in some cases, I think that's more important than having
the same (wrong IMHO) behaviour in all cases.  Even if we risk running an extra couple of
unnecessary tests due to concurrency issues when the fail counter is triggered, that's got
to be better than running all the tests, or turning off the rerun feature.  But I haven't
read that code, so I can't really suggest anything concrete.  I gather that this all has to
do with the fact that there is currently a separate rerun phase after the main testing phase,
so it might take a lot of refactoring to make the sort of change I have in mind.

In any case, what's implemented now is not good, because even though surefire-junit4 now in
fact does run flaky tests multiple times, the XML file does not reflect this - it only reports
the first run (a failure), and ignores the second run (which passed).  (This is inferred from
the logging in my real project's tests.)

Not to mention the fact that the description here: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
doesn't seem to describe what actually happens when both options are set (as I said in the
original report above). 

I have had to disable the fail-fast feature to get flaky reruns to work as they should, which
is a real shame because these two options would really complement each other.


> skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
> ------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1219
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 2.19.1
>            Reporter: Sean Flanigan
>         Attachments: SUREFIRE-1219-tests.zip
>
>
> According to https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
"failed tests within re-run phase are not included in skipAfterFailureCount". For instance,
if skipAfterFailureCount is 1, but every test passes on the second attempt, the tests are
"flaky" but not "failures".  This should be reflected in summaries, in XML result files and
in fail fast (skipAfterFailureCount), but this is not the case.
> If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and skipAfterFailureCount=5:
> 1. I would expect something like this to be printed on the console:
>   Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
> 2. The XML report should list <flakyFailure>s but no <failures>.
> 3. All tests should be run, not just 5 of them, in other words skipAfterFailureCount
should not count the flaky tests as part of the failure count.
> I have some tests demonstrating the problem, which I will attach to this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message