maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tibor17 <...@git.apache.org>
Subject [GitHub] maven-surefire pull request: [SUREFIRE-1202] Allow rerunFailingTes...
Date Fri, 04 Dec 2015 11:35:09 GMT
Github user Tibor17 commented on the pull request:

    https://github.com/apache/maven-surefire/pull/108#issuecomment-161945817
  
    @seanf 
    >>Are you saying you did try to build an IT for this?
    reuseForks=false would not work in 2.19 feature.
    Initially skipAfterFailureCount was not count because it was boolean and therefore rerunFailingTestsCount
did not make sense.
    Now it may have sense.
    
    >>What counts would you consider realistic?
    Maybe 5 test classes or more with Thread.sleep() 3 seconds, forkCount=2, rerunFailingTestsCount
= 3 and skipAfterFailureCount=2. There should be two tests. One should work with #failures
< rerunFailingTestsCount (all 5 test classes passed with flaky tests), and second test
in IT where #failures > rerunFailingTestsCount (totally maybe only 3 or 4 test classes
run and two failed). Each class has one method.
    The classes are ordered on alphabetical order and forked.
    That's my idea but you can improve it of course and the system properties can be used
in order to force tests to fail e.g.
    ```
     @Test test() {
      if (sys-prop) Assert.fail();
    }
    ```
    Let's see how long sleep() should be in the tests..
    
    >>I'm still assuming failureCount is meant to count failed tests, not bad
    runs. You didn't answer my question about that (and the docs don't say),
    but I do think it is better to count tests, as long as it is documented.
    
    If the rerun is enabled, the flaky tests (I mean those bad tests < rerunFailingTestsCount)
should not be a subject to the feature with skipAfterFailureCount. I don't remember the code
in detail. We need the IT to confirm both features may work together.
    
    >>As I said, there should be no race condition if failureCount counts
    (completely) failed tests, assuming the runs for each test are serialised.
    Race condition is between forks. In extremely short test method race condition may happen
and the failure count may grow over expected skipAfterFailureCount .



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message