maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Subach (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SUREFIRE-1470) Unexpected report result when using rerunFailingTestsCount with groups filter in Surefire
Date Thu, 29 Mar 2018 16:25:00 GMT

    [ https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419292#comment-16419292
] 

Alexey Subach commented on SUREFIRE-1470:
-----------------------------------------

I think the main goal is to have report results unified. The approach you suggested seems
ok. I will see if I am able to find some time to dig into the code. Thanks for your help!
Meanwhile the TeamCity team reported back that they only parse XML output, and according to
my tests the XML output is the same in all cases, maybe the problem is in XML output being
rewritten or something else. This also needs to be investigated.

> Unexpected report result when using rerunFailingTestsCount with groups filter in Surefire
> -----------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1470
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Junit 4.x support, Maven Surefire Plugin
>            Reporter: Alexey Subach
>            Priority: Major
>         Attachments: test.zip
>
>
> When the following plugin configuration is used:
> {code:java}
> <plugin>
>     <groupId>org.apache.maven.plugins</groupId>
>     <artifactId>maven-surefire-plugin</artifactId>
>     <version>${surefire.version}</version>
>     <configuration>
>         <includes>
>             <include>**/*.java</include>
>         </includes>
>         <groups>TestGroup</groups>
>         <rerunFailingTestsCount>1</rerunFailingTestsCount>
>     </configuration>
> </plugin>{code}
> And there is a test that fails first time it's run, the resultant *txt* report is unexpected
and only contains the result of last rerun.
> Suppose we have the following test class:
> {code:java}
> @Category(TestGroup.class)
> public class CustomTest {
>     @Test
>     public void alwaysPassingTest() {
>         Assert.assertEquals(0, 0);
>     }
>     private static volatile int x = 0;
>     @Test
>     public synchronized void secondAttemptPassingTest() throws InterruptedException {
>         Thread.sleep(5000);
>         if (x++ == 0) {
>             Assert.fail();
>         }
>     }
> }
> {code}
> The resultant xml report is all right, but the txt one is
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in CustomTest
> {code}
> Which is basically incorrect. The *txt* file is regenerated several times during the
test run.
> The issue is present when groups filter (either exclusion or inclusion) is present. When
there is no group filter and only rerunFailingTestsCount is configured, then the resultant
report is the following one:
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<<
FAILURE! - in CustomTest
> secondAttemptPassingTest(CustomTest)  Time elapsed: 0.009 s  <<< FAILURE!
> java.lang.AssertionError
> 	at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
> {code}
> When no group filter is configured, Junit4Provider instead of JunitCore provider is picked
up and they seem to handle those cases differently.
> The situation is worsened by the fact that TeamCity is not able to parse the results
correctly when the groups are configured along with rerunFailingTestsCount and it marks the
tests that passed after rerun as failed, although the overall Maven result is SUCCESS in both
cases.
> The simple Maven project to reproduce the issue is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message