lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Annotation for "run this test, but don't fail build if it fails" ?
Date Fri, 04 May 2012 22:29:45 GMT

Dawid:

With the new test runner you created, would it be possible to setup an 
annotation that we could use instead to indicate that a test should in 
fact be run, and if it fails, include the failure info in the build 
report, but do not fail the build?

I'm thinking in particular about some of the test that multi-threaded 
tests that are currently marked @Ignore or @AwaitsFix because they 
sporadically fail on jenkins in our jail -- but that people haven't been 
able to reproduce consistently on local dev machines (or that some people 
have been able to reproduce, but not the people who understand the 
tests/code well enough to try and fix)

As it stands right now, if somone wants to try and fix a complicated test 
that's disabled, they have to make a guess at the fix, un-@Ignore, then 
watch the next few/several builds patiently to see if / how-often it 
fails, then commit the @Ignore back, and repeat.

If we could leave these tests running on every build, then we could at 
least monitor the relative frequency of the failures -- ie: "last week 
testFoo failed in 10% of the builds, this week it fails in every 
build, so somebody definiteily broke something" or "last week testFoor 
failed in 10% of the builds, and after my attempted hardening it only 
fails in 5% of the builds so i may be on to something."

what do folks think?

-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message