lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@gmail.com>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sat, 05 May 2012 07:58:52 GMT
Not a problem. Ill be at work on monday. Can you file an issue an assign it
to me please? Im currently on mobile only. Dawid
On May 5, 2012 5:08 AM, "Yonik Seeley" <yonik@lucidimagination.com> wrote:

> On Fri, May 4, 2012 at 6:29 PM, Chris Hostetter
> <hossman_lucene@fucit.org> wrote:
> >
> > Dawid:
> >
> > With the new test runner you created, would it be possible to setup an
> > annotation that we could use instead to indicate that a test should in
> fact
> > be run, and if it fails, include the failure info in the build report,
> but
> > do not fail the build?
> >
> > I'm thinking in particular about some of the test that multi-threaded
> tests
> > that are currently marked @Ignore or @AwaitsFix because they sporadically
> > fail on jenkins in our jail -- but that people haven't been able to
> > reproduce consistently on local dev machines (or that some people have
> been
> > able to reproduce, but not the people who understand the tests/code well
> > enough to try and fix)
> >
> > As it stands right now, if somone wants to try and fix a complicated test
> > that's disabled, they have to make a guess at the fix, un-@Ignore, then
> > watch the next few/several builds patiently to see if / how-often it
> fails,
> > then commit the @Ignore back, and repeat.
> >
> > If we could leave these tests running on every build, then we could at
> least
> > monitor the relative frequency of the failures -- ie: "last week testFoo
> > failed in 10% of the builds, this week it fails in every build, so
> somebody
> > definiteily broke something" or "last week testFoor failed in 10% of the
> > builds, and after my attempted hardening it only fails in 5% of the
> builds
> > so i may be on to something."
> >
> > what do folks think?
>
> +1
>
> Something like this is definitely needed.
> Some of the Solr tests that spin up multiple JVMs are particularly
> tough to get 100% bullet-proof on all platforms (esp this freebsd
> jail) and there is a lot of information in tests that occasionally
> fail (esp if said tests may be the *only* tests we have for certain
> functionalities).
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Mime
View raw message