lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: Failing tests aka. "what's the point of running them?"
Date Fri, 09 Nov 2012 15:11:28 GMT
+1

I mean, yes, I would like to see any test failures addressed quickly, but 
for any tests that fail chronically, it makes sense to both disable/ignore 
them as well as make sure that "blocker" Jira's get filed for them.

Personally, I'd like to see Jira's for all test failure errors so that 
people can easily search Jira for any failure message, hang, etc., including 
reasonably detailed narrative to explain how the failure occurred and how it 
was tracked down and the nature of the fix so that future failures can be 
fixed more promptly and by more people. Leave enough info that less senior 
community members can begin to learn what it takes to fix test failures.

As things stand, I don't have a ghost of a chance at looking at any of these 
test failures - the whole test infrastructure is a black box with such 
complexity that I can fathom only the simplest of tests. If I have 
difficulty with this, maybe there are others who would benefit as well from 
a greater sharing of the expertise needed to track down these chronic and 
seemingly mysterious failures. I mean, the expertise to even LOOK at some of 
these failures is in the heads and hands of too small a set of individuals.

And maybe the list of failure modes is also indicative of the lack of a rich 
enough test infrastructure at the application level, especially when dealing 
with timing issues, let alone the vagaries of Java and individual JVM 
idiosyncrasies . The irony is that sometimes we spend more time trying to 
get the tests to pass on timing issues than to put more stress of code to 
expose more timing problems.

Oh, and if some JVM's are failing chronically, tag those JVM's as 
"unsupported" until sufficient testing and testing expertise is available to 
get things fixed. File detailed Jira's as well, as above, except NOT as 
blockers. I mean, let's get the chronic failures under control on the "main" 
supported platforms before expanding the supported environments - "supported 
but with significant and chronic test failures" should not be supported.

-- Jack Krupansky

-----Original Message----- 
From: Simon Willnauer
Sent: Friday, November 09, 2012 2:36 AM
To: dev@lucene.apache.org
Subject: Failing tests aka. "what's the point of running them?"

hey folks,

I know yonik and mark had a long time power outage so I don't want to
blame anybody here but we need to fix those test failures.
Really when you look at those jenkins jobs:

http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Windows/
http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/
https://builds.apache.org/computer/lucene/
https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/

its really funny if that'd be a joke but it isn't. If I'd be a new
contributor I'd be scared as sh** when I subscribe to the mailing
list. It's also not a good advertisement for us either. Yet, even
further it makes me miss failures I might have caused since the amount
of failure mails don't encourage to look at them since its the same
tests that fail over and over again no matter what code I commit. I
can already hear somebody saying "why don't you fix it" - well fair
enough but this project is massive and we are a large committer base
and I don't see myself fix the code I have never ever touched. Anyhow,
I really ask myself what is the point of running these tests,
specifically the solr ones, if the fail over and over again and nobody
cares? Even if folks care they don't get fixed and this project as
more committers than yonik and mark. Its really a bad sign if we are
at the point where we rely on 2 people to fix tests on a stable
branch...
I'd really like to hear how people want to address this, I mean it
would be just fair to disable the tests until somebody has the
patience / time to fix them we can / should make it blockers for a
release. Really a jenkins mail should be the exception not the
rule...I also think if the FreeBSD jenkins black hole stuff is a
problem for the tests then lets add a @BlackHoleProne annotation and
only run that on linux? I really don't care how we fix it but if we
don't have a solution by the end of next week I will add @Ignore to
all of them that failed in the last 2 weeks.

Sorry I got so frustrated about this - this is really bad press here!

simon

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message