maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Kriegisch <>
Subject Re: Extend surefire to rerun failing tests
Date Wed, 28 May 2014 14:19:52 GMT

I have not tried, Kristian, but it looks promising. Both your link and the randomised testing
one I have forwarded to my old Scrum team (I am no longer coaching them, thus unable to access
their code base and try for myself).

Thank you. :-)
Alexander Kriegisch

> Am 28.05.2014 um 09:50 schrieb Kristian Rosenvold <>:
> Does this fit the bill for you ?
> Kristian
> 2014-05-28 9:14 GMT+02:00 Alexander Kriegisch <>:
>> --
>> Alexander Kriegisch
>>> Am 28.05.2014 um 05:13 schrieb Benson Margulies <>:
>>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <>wrote:
>>>> I am an intern at Google. The first step of my internship project is to add
>>>> the ability to Maven to automatically rerun failing tests a few times, to
>>>> see if they ever pass in any of the reruns. It is useful because in many
>>>> cases a test fails because it is flaky, not because there is a bug in the
>>>> new source code change.
>>> I entirely disagree. 99.9999% of programs are completely deterministic. If
>>> you feed them the same inputs, you get the same output.
>> Well, in my previous project (I am an Agile Coach) my teams were working on a distributed,
multi-threaded system and around 10% of all unit tests and next to all integration tests were
affected by the exception you mentioned:
>>> The exceptions are tests that use threads, and it's quite hard to do a good
>>> job writing them.
>> One can argue that several of the unit tests were not real unit tests because they
mocked some, but not all of the infrastructure they needed for running. Anyway, they were
running in Surefire (the integration tests in Failsafe), and some lf them were flaky.
>>> Just 'rerunning a few times' does not lead to useful
>>> information in my view.
>> Only the information that indeed they are flaky and a rough percentage of cases in
which they fail, which would be rather dependend on the test environment though. Better than
nothing information-wise. So far, so good.
>> Having said all that, I still agree to Benson that a failing test should just fail,
period. I do not want developers to think, "it is only failing only in 5% of all runs, so
it is fine / low priority to fix". OTOH I understand Qingzhou from a perspective of wishing
to "test the test" or rather wishing to harden a flaky test into a stable one, either by fixing
the test or the threading bug(s) it exposes. Benson's suggestion below looks promising though,
so thanks for that.
>>> if you want random testing, there's a fine alternative at
>>>, and it has a fine
>>> maven plugin. It can already 'rerun your tests a few times' very easily.
>>> I wonder what the more involved surefire maintainers think, but as a PMC
>>> member I'd be -1 on this proposal.
>>>> We think the right way to achieve this is to modify surefire plugin of
>>>> maven. We want to add it as a part of the configuration of surefire, so
>>>> users can decide whether they want to enable this feature, and how many
>>>> times they want to rerun failing tests. We plan to open-source our
>>>> contribution, and hopefully can merge our code into surefire master branch
>>>> in the end. Therefore,  we are wondering do you have any
>>>> comments/suggestions/opinions regarding this? We appreciate any input.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message