lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gerlowski <>
Subject Re: Test failures are out of control......
Date Wed, 21 Feb 2018 23:08:48 GMT
I don't have strong opinions about what we do with our existing flaky
tests.  I think re-running failures before commit might theoretically
catch more bugs than ignoring the test outright, but with all the
noise and how standard it is to need to rerun tests I'd be surprised
if the numbers are all that different.

Where I see some potential common ground though is in preventing new
flaky tests.  And in the long run, I think what we do to prevent new
flakes is going to be much more important than how we handle the
BadApples we have at this particular instant.  If we can put our
finger in the dam, the existing flakiness becomes much easier to put a
dent in.

I'm curious what you guys had in mind when you mentioned preventing
new flaky tests from popping up.  What are our options for "enforcing"
that?  Were you imagining reopening JIRAs and asking the original
committer to investigate?  Or outright reverting commits that
introduce flaky tests?  Or something in between (like disabling
features with flaky tests prior to releases)?



On Wed, Feb 21, 2018 at 6:03 PM, Uwe Schindler <> wrote:
> Hi,
>> Flakey Test Problems:
>> a) Flakey tests create so much noise that people no longer pay
>> attention to the automated reporting via email.
>> b) When running unit tests manually before a commit (i.e. "ant test")
>> a flakey test can fail.
>> Solutions:
>> We cloud fix (a) by marking as flakey and having a new target
>> "non-flakey" that is run by the jenkins jobs that are currently run
>> continuously.
> We have a solution for this already: Mark all those tests with @AwaitsFix or @BadApple
> By default those aren't executed in Jenkins runs and also not for developers, but devs
can enable/disable them using -Dtests.awaitsfix=true and -Dtests.badapples=true:
>      [help] # Test groups. ----------------------------------------------------
>      [help] #
>      [help] # test groups can be enabled or disabled (true/false). Default
>      [help] # value provided below in [brackets].
>      [help]
>      [help] ant -Dtests.nightly=[false]   - nightly test group (@Nightly)
>      [help] ant -Dtests.weekly=[false]    - weekly tests (@Weekly)
>      [help] ant -Dtests.awaitsfix=[false] - known issue (@AwaitsFix)
>      [help] ant -Dtests.slow=[true]       - slow tests (@Slow)
> We can of course also make a weekly jenkins jobs that enables those tests on Jenkins
only weekly (like nightly stuff). We have "tests.badapples" and "tests.awaitsfix" - I don't
know what's the difference between both.
> So we have 2 options to classify tests, let's choose one and apply it to all Flakey tests!
> Uwe
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message