lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@cs.put.poznan.pl>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sun, 06 May 2012 18:39:42 GMT
So, I started thinking about it -- I can implement something that will
report failures (much like we do right now) it's quite tricky to fit
it into the reporting system and continuous integration system. Here's
why -- if a test doesn't fail then its output (sysout/syserrs) are not
currently printed (to provide a cleaner view of what's been executed).
Verbose log is on disk but it'd have to be scanned by hand (and copied
as a build artifact). Yet another problem is that jenkins wouldn't
_fail_ on such pseudo-failures because the set of JUnit statuses is
not extensible (it'd be something like FAILED+IGNORE) so we'd need to
either go with IGNORED, ASSUMPTION_IGNORED or SUCCESS, none of which
are a good match, really. ASSUMPTION_IGNORED status is probably most
convenient here because of how it can be technically propagated back
to JUnit.

Any ideas? Hoss -- how do you envision "monitoring" of these tests? Manually?

Dawid

> If we could leave these tests running on every build, then we could at least
> monitor the relative frequency of the failures -- ie: "last week testFoo
> failed in 10% of the builds, this week it fails in every build, so somebody
> definiteily broke something" or "last week testFoor failed in 10% of the
> builds, and after my attempted hardening it only fails in 5% of the builds
> so i may be on to something."
>
> what do folks think?
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message