lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sun, 06 May 2012 19:56:59 GMT
On Sun, May 6, 2012 at 3:38 PM, Dawid Weiss
<dawid.weiss@cs.put.poznan.pl> wrote:
> I also admit I've never seen anything like
> this -- a suite of tests with an allowed failure ratio over time and a
> threshold that would trigger a warning...

Not so much an "allowed" failure rate... more of "it fails sometimes
and no one has had the time to try to get it to pass with a greater
percentage of time".
And even when people put effort into get it to pass more often, it's
still not 100%.

As those tests exist now, there are a few choices
a) turn them off (this is bad because it seriously decreases coverage)
b) somehow deal with the intermittent failures

Given that we're not running on a realtime system, the fact that many
higher level tests have timing and scheduling dependencies means that
we will never achieve a 100% pass rate on such tests.

> These are weird tests if they allow for a (predictable?) failure from
> time to time. I don't say it's a bad concept, but I think unit tests
> may not be a good framework for handling this.

Yeah, these aren't really unit tests.  Should we try to move them
somewhere else?  Or run them separately and email the results to a
different list?

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message