lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@gmail.com>
Subject Re: Annotation for "run this test, but don't fail build if it fails" ?
Date Sat, 05 May 2012 09:49:16 GMT
One other way to do it that is already implemented is to run full tests
without failing on failures and only touch some marker file to fail at the
end. Ant test-help gives a hint on how to run tests this way currently.
Dont know how it'd play with others - speak up.
On May 5, 2012 11:25 AM, "Mikhail Khludnev" <mkhludnev@griddynamics.com>
wrote:

> It's a pretty useful technique especially from CI perspective. I use it
> via JUnit's assumptions. Failed assumption is shown as an ignored test.
>
> On Sat, May 5, 2012 at 2:29 AM, Chris Hostetter <hossman_lucene@fucit.org>wrote:
>
>>
>> Dawid:
>>
>> With the new test runner you created, would it be possible to setup an
>> annotation that we could use instead to indicate that a test should in fact
>> be run, and if it fails, include the failure info in the build report, but
>> do not fail the build?
>>
>> I'm thinking in particular about some of the test that multi-threaded
>> tests that are currently marked @Ignore or @AwaitsFix because they
>> sporadically fail on jenkins in our jail -- but that people haven't been
>> able to reproduce consistently on local dev machines (or that some people
>> have been able to reproduce, but not the people who understand the
>> tests/code well enough to try and fix)
>>
>> As it stands right now, if somone wants to try and fix a complicated test
>> that's disabled, they have to make a guess at the fix, un-@Ignore, then
>> watch the next few/several builds patiently to see if / how-often it fails,
>> then commit the @Ignore back, and repeat.
>>
>> If we could leave these tests running on every build, then we could at
>> least monitor the relative frequency of the failures -- ie: "last week
>> testFoo failed in 10% of the builds, this week it fails in every build, so
>> somebody definiteily broke something" or "last week testFoor failed in 10%
>> of the builds, and after my attempted hardening it only fails in 5% of the
>> builds so i may be on to something."
>>
>> what do folks think?
>>
>> -Hoss
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.**org<dev-unsubscribe@lucene.apache.org>
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
> --
> Obviousnessly Yours
> Captain
>
> <http://www.griddynamics.com>
> <mkhludnev@griddynamics.com>
>
>

Mime
View raw message