lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sat, 05 May 2012 09:24:33 GMT
It's a pretty useful technique especially from CI perspective. I use it via
JUnit's assumptions. Failed assumption is shown as an ignored test.

On Sat, May 5, 2012 at 2:29 AM, Chris Hostetter <hossman_lucene@fucit.org>wrote:

>
> Dawid:
>
> With the new test runner you created, would it be possible to setup an
> annotation that we could use instead to indicate that a test should in fact
> be run, and if it fails, include the failure info in the build report, but
> do not fail the build?
>
> I'm thinking in particular about some of the test that multi-threaded
> tests that are currently marked @Ignore or @AwaitsFix because they
> sporadically fail on jenkins in our jail -- but that people haven't been
> able to reproduce consistently on local dev machines (or that some people
> have been able to reproduce, but not the people who understand the
> tests/code well enough to try and fix)
>
> As it stands right now, if somone wants to try and fix a complicated test
> that's disabled, they have to make a guess at the fix, un-@Ignore, then
> watch the next few/several builds patiently to see if / how-often it fails,
> then commit the @Ignore back, and repeat.
>
> If we could leave these tests running on every build, then we could at
> least monitor the relative frequency of the failures -- ie: "last week
> testFoo failed in 10% of the builds, this week it fails in every build, so
> somebody definiteily broke something" or "last week testFoor failed in 10%
> of the builds, and after my attempted hardening it only fails in 5% of the
> builds so i may be on to something."
>
> what do folks think?
>
> -Hoss
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.**org<dev-unsubscribe@lucene.apache.org>
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Obviousnessly Yours
Captain

<http://www.griddynamics.com>
<mkhludnev@griddynamics.com>

Mime
View raw message