lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sun, 06 May 2012 19:56:59 GMT
On Sun, May 6, 2012 at 3:38 PM, Dawid Weiss
<> wrote:
> I also admit I've never seen anything like
> this -- a suite of tests with an allowed failure ratio over time and a
> threshold that would trigger a warning...

Not so much an "allowed" failure rate... more of "it fails sometimes
and no one has had the time to try to get it to pass with a greater
percentage of time".
And even when people put effort into get it to pass more often, it's
still not 100%.

As those tests exist now, there are a few choices
a) turn them off (this is bad because it seriously decreases coverage)
b) somehow deal with the intermittent failures

Given that we're not running on a realtime system, the fact that many
higher level tests have timing and scheduling dependencies means that
we will never achieve a 100% pass rate on such tests.

> These are weird tests if they allow for a (predictable?) failure from
> time to time. I don't say it's a bad concept, but I think unit tests
> may not be a good framework for handling this.

Yeah, these aren't really unit tests.  Should we try to move them
somewhere else?  Or run them separately and email the results to a
different list?

-Yonik - Lucene/Solr Open Source Search Conference.
Boston May 7-10

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message