lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sun, 13 May 2012 08:28:29 GMT
> jenkins "run all tests continuously" build target that only runs the
> @AwaitsFix group, and overrides haltonfailure when calling the junit
> macro. (which would save us an extra jenkins run)

Yep, doable.

> decreased the likelyhood of failure, but didn't completely fix the
> problem -- a test might fail an average of 1/5 times, and someone might
> start working on improving them, and reduce the failure rate to 1/20

You're making them stochastic which is an interesting direction for
research :) I mean, typically the direction would be to make (unit)
tests predictable and repeatable given the same starting conditions.
Whether this is always possible is a different discussion, I
understand it.

> evaluating lucene/solr and looking at the tests could see in a very clear
> way "testDoSomeStuffWithFeatureXandFeatureYTogether() fails 100%
> of the time, so i probably shouldn't use X and Y together"


> build) and just generated the normal junit test output, with the pretty
> graph showing all the failing @AwaitsFix tests in red, so people could

Don't know about the pretty graph but if jenkins includes individual
test tracking (and I think it does) then just running those tests
without causing a build failure would give you a time-based overview
of their execution status. You would still need to know which tests to
look at because I don't think jenkins tracks "frequently failing"
tests (the way Atlassian's Bamboo does for example).

It would be definitely possible to build an infrastructure that would
generate the above from test logs we produce but I won't have the time
to look into it in the nearest future given the number of issues
assigned to me in Lucene (and elsewhere...). We can start small by
adding that "no-fail" test run - this would still leave a track of
failed logs so that when somebody has the time to write up that view
over time they have some data to work with. This would be an
interesting plugin to jenkins in general... perhaps they have a GSoC
student or something and we could suggest it to them?


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message