lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <>
Subject RE: Test failures are out of control......
Date Wed, 21 Feb 2018 23:03:22 GMT

> Flakey Test Problems:
> a) Flakey tests create so much noise that people no longer pay
> attention to the automated reporting via email.
> b) When running unit tests manually before a commit (i.e. "ant test")
> a flakey test can fail.
> Solutions:
> We cloud fix (a) by marking as flakey and having a new target
> "non-flakey" that is run by the jenkins jobs that are currently run
> continuously.

We have a solution for this already: Mark all those tests with @AwaitsFix or @BadApple
By default those aren't executed in Jenkins runs and also not for developers, but devs can
enable/disable them using -Dtests.awaitsfix=true and -Dtests.badapples=true:

     [help] # Test groups. ----------------------------------------------------
     [help] #
     [help] # test groups can be enabled or disabled (true/false). Default
     [help] # value provided below in [brackets].
     [help] ant -Dtests.nightly=[false]   - nightly test group (@Nightly)
     [help] ant -Dtests.weekly=[false]    - weekly tests (@Weekly)
     [help] ant -Dtests.awaitsfix=[false] - known issue (@AwaitsFix)
     [help] ant -Dtests.slow=[true]       - slow tests (@Slow)

We can of course also make a weekly jenkins jobs that enables those tests on Jenkins only
weekly (like nightly stuff). We have "tests.badapples" and "tests.awaitsfix" - I don't know
what's the difference between both.

So we have 2 options to classify tests, let's choose one and apply it to all Flakey tests!


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message