lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sat, 12 May 2012 18:55:13 GMT

: This is doable by enabling/disabling test groups. A new build plan
: would need to be created that would do:
: 
: ant -Dtests.haltonfailure=false -Dtests.awaitsfix=true
: -Dtests.unstable=true test

right ...that's an idea that came up the other day when i was talking to 
simon at revolution

another idea that came up this morning talking with rmuir is to have a new 
ant target "ie: test-needs-fix" that could be run as part of the same 
jenkins "run all tests continuously" build target that only runs the 
@AwaitsFix group, and overrides haltonfailure when calling the junit 
macro. (which would save us an extra jenkins run)

: Doable, but honestly this seems like more work (scripts for collecting
: stats, test groups are trivial) than trying to fix those two or three
: tests that fail?

but there are 3 distinct (in my mind) issues here tthat all of this helps 
address:

1) those "two or three" tests that fail and should be fixed ... i agree, 
we should fix them, but when you can't reproduce the failures at all, it's 
really hard to iterate and figure out what needs fixed.  if jenkins is 
where they fail, we need a way for jenkins to run them so we can see if 
our attempted fixes work.

2) all of that is double painful when they only fail sporadically, it 
might take a week of constant jenkins testing to discover that your fix 
decreased the likelyhood of failure, but didn't completely fix the 
problem -- a test might fail an average of 1/5 times, and someone might 
start working on improving them, and reduce the failure rate to 1/20 
times - they need to be running constantly (with some way to review the 
rate of failure) to make progress

3) we have some tests that demonstrate bugs no one has ever fixed because 
they are too hard to fix, or we don't have a good solution for them.  
some of these tests are commited but @Ignored, some are commited but 
commented out, some are sitting in patches in jira waiting for the patch 
to be expanded to include the code -- it would be nice if all of those 
tests could be commited, and uncommented, and run on every build, failing 
100% of the time, so the known weaknesses in the code that now one has 
time/energy/ideas on how to fix would be more publicly visible, and people 
evaluating lucene/solr and looking at the tests could see in a very clear 
way "testDoSomeStuffWithFeatureXandFeatureYTogether() fails 100% 
of the time, so i probably shouldn't use X and Y together"


I think if running "ant test test-needs-fix" executed all the normal tests 
(which fail the build) as well as all @AwaitsFix (which wouldn't fail the 
build) and just generated the normal junit test output, with the pretty 
graph showing all the failing @AwaitsFix tests in red, so people could 
realisticly see "some stuff doesn't work, and some of that stuff doesn't 
work sporadically" then we would benefit us in multiple ways (more data to 
help fix sporadically failing tests, more open honesty about what doesn't 
work) and when people fix @AwaitsFix tests, they just remove the 
annotation.


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message