maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tibor Digana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SUREFIRE-1264) Some tests can be lost when running in parallel with parameterized tests
Date Tue, 27 Jun 2017 05:20:00 GMT

    [ https://issues.apache.org/jira/browse/SUREFIRE-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064283#comment-16064283
] 

Tibor Digana commented on SUREFIRE-1264:
----------------------------------------

[~tlemeur]
I changed your POM. I do not see problem. What's wrong?
Do not use plugin in dependencies.
Use {{surefire-junit47}} provider in dependencies embedded within plugin and not in project/dependencies;
see the documentation [provider for plugin|http://maven.apache.org/surefire/maven-surefire-plugin/examples/providers.html].
{code}
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.CTest
Start: CTest.methodC1[p0]
End:   CTest.methodC1[p0]
Start: CTest.methodC2[p0]
End:   CTest.methodC2[p0]
Start: CTest.methodC1[p1]
End:   CTest.methodC1[p1]
Start: CTest.methodC2[p1]
End:   CTest.methodC2[p1]
Start: CTest.methodC1[p2]
End:   CTest.methodC1[p2]
Start: CTest.methodC2[p2]
End:   CTest.methodC2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.301 s - in com.appnexus.viewability.core.surefireJunitTests.CTest
[INFO] Running com.appnexus.viewability.core.surefireJunitTests.ATest
Start: ATest.methodA2[p0]
End:   ATest.methodA2[p0]
Start: ATest.methodA1[p0]
End:   ATest.methodA1[p0]
Start: ATest.methodA1[p1]
End:   ATest.methodA1[p1]
Start: ATest.methodA2[p1]
End:   ATest.methodA2[p1]
Start: ATest.methodA2[p2]
End:   ATest.methodA2[p2]
Start: ATest.methodA1[p2]
End:   ATest.methodA1[p2]
[ERROR] Tests run: 6, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.205 s <<<
FAILURE! - in com.appnexus.viewability.core.surefireJunitTests.ATest
[ERROR] methodA1[p0](com.appnexus.viewability.core.surefireJunitTests.ATest)  Time elapsed:
0.105 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p0]
        at com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[ERROR] methodA1[p1](com.appnexus.viewability.core.surefireJunitTests.ATest)  Time elapsed:
0.1 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p1]
        at com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[ERROR] methodA1[p2](com.appnexus.viewability.core.surefireJunitTests.ATest)  Time elapsed:
0.1 s  <<< FAILURE!
java.lang.AssertionError: Failing test: ATest.methodA1[p2]
        at com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1(ATest.java:14)

[INFO] Running com.appnexus.viewability.core.surefireJunitTests.BTest
Start: BTest.methodB1[p0]
End:   BTest.methodB1[p0]
Start: BTest.methodB2[p0]
End:   BTest.methodB2[p0]
Start: BTest.methodB1[p1]
End:   BTest.methodB1[p1]
Start: BTest.methodB2[p1]
End:   BTest.methodB2[p1]
Start: BTest.methodB1[p2]
End:   BTest.methodB1[p2]
Start: BTest.methodB2[p2]
End:   BTest.methodB2[p2]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.206 s - in com.appnexus.viewability.core.surefireJunitTests.BTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p0]
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p1]
[ERROR]   ATest.methodA1:14 Failing test: ATest.methodA1[p2]
[INFO]
[ERROR] Tests run: 18, Failures: 3, Errors: 0, Skipped: 0
[INFO]
{code}

> Some tests can be lost when running in parallel with parameterized tests
> ------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1264
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1264
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Junit 4.7+ (parallel) support
>    Affects Versions: 2.19.1
>         Environment: Maven 3.3.9
> Java 1.8.0_45 (Oracle)
> System: OS X
> Reproduced on Linux too, with both OpenJDK 7 and Java 7 from Oracle.
>            Reporter: Jean-Luc Derrien
>            Assignee: Tibor Digana
>             Fix For: 2.20.1
>
>
> Hello,
> It appears some tests can be lost when using the parallel mode with parameterized tests.
Here is a [small project|https://github.com/jderrien/surefire-junit-tests/tree/simple-1] I've
used to try to reproduce the issue we have seen.
> This is not something that happens on every run, but it's quite frequent.
> With this loop, the problem should appear in less than 2 minutes, maybe on the first
run when (un)lucky:
> {code}
> time while true; do mvn clean test > last.log ; tail -25 last.log ; if [ "$(grep -c
'Tests run: 12' last.log)" == "0" ]; then break; fi ; done
> {code}
> Normal run:
> {noformat}
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime
= 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime
= 1 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime
= 2 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime
= 2 => stop
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime
= 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime
= 5 => stop
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.006 sec - in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in [p2]
> Results :
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> When tests have been lost (here one test lost according to the output):
> {noformat}
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running [p2]
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p0] - p0 - sleeptime
= 5 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p2] - p2 - sleeptime
= 1 => start
> com.appnexus.viewability.core.surefireJunitTests.ATest.methodA1[p1] - p1 - sleeptime
= 2 => start
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in [p2]
> Running [p2]
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in [p2]
> Results :
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
> Note: there are 3 test classes and 18 tests on the [master branch|https://github.com/jderrien/surefire-junit-tests/tree/master]
and it's even more frequent/easy to reproduce.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message