maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Falko Modler (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SUREFIRE-1728) maven.test.failure.ignore: differentiate between test failure and timeout
Date Sun, 01 Dec 2019 22:56:00 GMT

    [ https://issues.apache.org/jira/browse/SUREFIRE-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985711#comment-16985711
] 

Falko Modler commented on SUREFIRE-1728:
----------------------------------------

{quote}
you want to express an error or failure message in the XML or TXT report.
{quote}
Not necessarily. I mean yes, this would be one option but I was thinking more whether the
plugin could just cause a build failure like _without_ {{maven.test.failure.ignore}}.

In short:
# {{maven.test.failure.ignore=true}} + timeout: build failure, test results up to the point
of the timeout can be parsed from the reports
# {{maven.test.failure.ignore=true}} + test failure: _no_ build failure, all results can be
parsed from the reports
# {{maven.test.failure.ignore=false}} + timeout: same as 1.
# {{maven.test.failure.ignore=false}} + test failure: build failure, test results can be parsed
from the reports

In case scenario/variant 1. is too much of a "breaking change", the value for {{maven.test.failure.ignore}}
could be extended to something like {{true/all}} | {{failures}} | {{false/none}}.
Just a thought...

> maven.test.failure.ignore: differentiate between test failure and timeout
> -------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1728
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1728
>             Project: Maven Surefire
>          Issue Type: Improvement
>    Affects Versions: 2.22.2, 3.0.0-M3
>         Environment: Maven 3.6.2
>            Reporter: Falko Modler
>            Assignee: Tibor Digana
>            Priority: Major
>             Fix For: 3.0.0-M5
>
>
> On a build server like Jenkins, people typically set {{-Dmaven.test.failure.ignore=true}}
to get the maximum number of test results instead of failing the build after the first module
with test failures.
> Unfortunately, timeouts are also ignored when this property is activated, leaving the
Jenkins JUnit plugin no chance to detect that something went wrong (because a timeout is not
reported in a txt or xml report).
> See also [JENKINS-46553|https://issues.jenkins-ci.org/browse/JENKINS-46553].
> The two cases should be differentiated.
> Due to backward compatibility reasons, I am not sure whether it would be wise to simply
exclude timeout cases.
> One backward compatible solution might be to extend the value range of {{maven.test.failure.ignore}}
from just {{true}} XOR {{false}} to something like:
> {{true}}/{{all}} XOR {{failure}} XOR {{false}}.
> The alternative would be to introduce yet another property...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message