lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <>
Subject Re: being a good citizen is hard when you can't successfully run tests....
Date Mon, 17 Sep 2012 12:51:04 GMT
Hmmm, in my other digression I thought I was on this thread.

No disagreement at all with the "stability report" idea etc.

To address being able to run tests, does it make sense
to create a new annotation "PeskyTest" or something? It would
suit my use case, that people need to be able to reliably run all
reasonable tests without much pain before checking code in.
We could leave these tests running all the time for everyone, but add
a note to "How To Contribute" about  "feel free to check code in if
it passes all tests when you disable PeskyTest" (and tell them how).

By leaving in on by default, we'd get maximum test coverage on
different machines/environments/whatever, but give people a
good way to get the warm fuzzy that their new changes didn't
break tests.


On Mon, Sep 17, 2012 at 8:42 AM, Michael McCandless
<> wrote:
> I agree that a test that frequently fails, and does not get fixed, is
> nearly pointless: everybody ignores it so it's as if the test didn't
> exist.  And so it should be disabled.
> I say *nearly* because the failures are in fact useful to devs who do
> have the itch/time to debug/fix them.
> So I think we need some middle ground here, where the tests keep
> failing but only those that are interested in the failures see the
> notifications.  We need to switch from a "push" model (any failure is
> broadcast to everybody) to a "pull" model (those devs that want to
> debug the failures go and check the logs), for such tests.
> When someone wants to make sure their change didn't break something
> (Erick's original use case) then these tests should not run.
> I like Dawid's idea (a separate test plan that Jenkins runs with these
> "difficult" tests, and it wouldn't email dev on failure).
> Mike McCandless
> On Mon, Sep 17, 2012 at 7:58 AM, Robert Muir <> wrote:
>> On Sun, Sep 16, 2012 at 11:10 PM, Mark Miller <> wrote:
>>> I get value from this test - if it was disabled, I'd probably re-enable it.
>>> would be great if it didn't fail so much, but the type of fail tells me
>>> something.
>> That means the assert in question isnt important at all. I'll remove it.
>> Again my problem is the idea that having a failing build is "ok"
>> because certain types of failures "don't matter". If they dont matter
>> they should be removed.
>> It causes a ton of noise when people are lazy about tests in this way,
>> and it wastes a ton of peoples time. R
>> Remember every time one of these tests fails it sends an email, that I
>> must read (we don't yet have a way to put in the subject header its a
>> SOLR test fail versus a LUCENE one, or i'd filter the solr ones and
>> not be complaining as much).
>> --
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message