lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sat, 05 May 2012 03:07:34 GMT
On Fri, May 4, 2012 at 6:29 PM, Chris Hostetter
<hossman_lucene@fucit.org> wrote:
>
> Dawid:
>
> With the new test runner you created, would it be possible to setup an
> annotation that we could use instead to indicate that a test should in fact
> be run, and if it fails, include the failure info in the build report, but
> do not fail the build?
>
> I'm thinking in particular about some of the test that multi-threaded tests
> that are currently marked @Ignore or @AwaitsFix because they sporadically
> fail on jenkins in our jail -- but that people haven't been able to
> reproduce consistently on local dev machines (or that some people have been
> able to reproduce, but not the people who understand the tests/code well
> enough to try and fix)
>
> As it stands right now, if somone wants to try and fix a complicated test
> that's disabled, they have to make a guess at the fix, un-@Ignore, then
> watch the next few/several builds patiently to see if / how-often it fails,
> then commit the @Ignore back, and repeat.
>
> If we could leave these tests running on every build, then we could at least
> monitor the relative frequency of the failures -- ie: "last week testFoo
> failed in 10% of the builds, this week it fails in every build, so somebody
> definiteily broke something" or "last week testFoor failed in 10% of the
> builds, and after my attempted hardening it only fails in 5% of the builds
> so i may be on to something."
>
> what do folks think?

+1

Something like this is definitely needed.
Some of the Solr tests that spin up multiple JVMs are particularly
tough to get 100% bullet-proof on all platforms (esp this freebsd
jail) and there is a lot of information in tests that occasionally
fail (esp if said tests may be the *only* tests we have for certain
functionalities).

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message