maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristian Rosenvold <kristian.rosenv...@gmail.com>
Subject Re: Extend surefire to rerun failing tests
Date Wed, 28 May 2014 19:43:54 GMT
I would really like to see and understand properly how this would
collaborate with JUnit rules for retries. I suspect there might be
some interesting issues regarding reporting (i.e. can the existing
logic handle reporting of n different executions of a single test).
How does this explode i parallel junit is selected :) ?

Technically you can just pass a system property from your pom and into
the surefire fork with the number of retries, you can read this
proprety in the junit rule. I really think you should start with this
approach and flesh out the details from there. It might turn out this
is all you need, or that some other entirely unforseen issues pop up.

As for the actual property in the plugin, I really think we should see
where this leads before deciding on how to do this. On a similar note,
I know some people with large corporate clouds sometimes run a test on
10000 different nodes in parallel, to investigate if there is any kind
of 1/10000 type failures, just to be able to summarize in 20 seconds
instead of 10000 linear runs.

This might also be another one of those cases where we could consider
picking up user-written code from the test classpath; but I think I
need another year thinking about that :)

Kristian




2014-05-28 19:58 GMT+02:00 Jason van Zyl <jason@takari.io>:
> I personally don't see an issue with those feature additions, but Kristian does the lions
share of the work on the Surefire plugin so it would be his call. I would definitely support
the additions of those features and would help test.
>
> On May 28, 2014, at 1:46 PM, Qingzhou Luo <qingzhouluo@google.com> wrote:
>
>> Hi Jason,
>>
>> Thanks for your reply. We plan to first extend surefire, so users can put a boolean
parameter rerunFailintTests and an int parameter rerunFailintTestsCount, to tell surefire
whether to rerun failing tests immediately after they fail. If a test passes in any of those
reruns if will be marked as pass, but all the failing attempts will still be recorded and
may be presented to users in a different way.
>>
>> We plan to finish everything in next 1-2 months. So we wonder would you be willing
to accept our contribution in the end, and patch our code into surefire?
>>
>> Thanks,
>>
>> Qingzhou
>>
>>
>> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <jason@takari.io> wrote:
>>
>> On May 27, 2014, at 1:14 PM, Qingzhou Luo <qingzhouluo@google.com> wrote:
>>
>>> Hi,
>>>
>>> I am an intern at Google. The first step of my internship project is to add
>>> the ability to Maven to automatically rerun failing tests a few times, to
>>> see if they ever pass in any of the reruns. It is useful because in many
>>> cases a test fails because it is flaky, not because there is a bug in the
>>> new source code change.
>>>
>>
>> I think generally being able to re-run failed tests is useful. I use this feature
in the IDE all the time in a fail/fix/run/repeat scenario. If you're trying to identify flaky
tests I think that's useful especially if you can collect statistics. While In the long term
you may not want to rely on the re-running of failed tests that later succeed without alteration
as the basis for good testing, but it's a good identification tool, but a useful feature in
its own right.
>>
>>> We think the right way to achieve this is to modify surefire plugin of
>>> maven. We want to add it as a part of the configuration of surefire, so
>>> users can decide whether they want to enable this feature, and how many
>>> times they want to rerun failing tests. We plan to open-source our
>>> contribution, and hopefully can merge our code into surefire master branch
>>> in the end. Therefore,  we are wondering do you have any
>>> comments/suggestions/opinions regarding this? We appreciate any input.
>>>
>>
>> Maybe a simple mechanism where you store the failed tests in a file and then have
a flag to run only the failed tests. I think this would be very useful. Then possible a small,
configurable loop around the main execution of tests if you wanted to specify how many times
to run the tests.
>>
>>> Thank you very much!
>>>
>>> Regards,
>>>
>>> Qingzhou
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder,  Apache Maven
>> http://twitter.com/jvanzyl
>> http://twitter.com/takari_io
>> ---------------------------------------------------------
>>
>> The modern conservative is engaged in one of man's oldest exercises in moral philosophy;
that is,
>> the search for a superior moral justification for selfishness.
>>
>>  -- John Kenneth Galbraith
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder,  Apache Maven
> http://twitter.com/jvanzyl
> http://twitter.com/takari_io
> ---------------------------------------------------------
>
> Selfish deeds are the shortest path to self destruction.
>
>  -- The Seven Samuari, Akira Kurosawa
>
>
>
>
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message