Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of dawid.weiss@gmail.com
 designates 209.85.160.48 as permitted sender)
MIME-Version: 1.0
Sender: dawid.weiss@gmail.com
In-Reply-To: <alpine.DEB.2.00.1205071013420.11210@bester>
References: <alpine.DEB.2.00.1205041516210.3541@bester>
 <CAM21Rt-653h25DbsyY9suW82u01FQnf0wo+T7zEyBV6pMX3tjg@mail.gmail.com>
 <alpine.DEB.2.00.1205071013420.11210@bester>
From: Dawid Weiss <dawid.weiss@cs.put.poznan.pl>
Date: Wed, 9 May 2012 10:12:31 +0200
Message-ID: 
 <CAM21Rt9SRXLKpp-iaKFkhRiX4m5oH7rLw=y562nVxPc63XVmKA@mail.gmail.com>
Subject: Re: Annotation for "run this test,
 but don't fail build if it fails" ?
To: dev@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

> That was really the main question i had, as someone not very familiar wit=
h
> the internals of JUnit, is wether it was possible for our test runner to
> make the ultimate decision about the success/fail status of the entire
> run based on the annotations of the tests that fail/succed

There are two things that need to be distinguished:

1) the "runner" is what's passed to @RunWith(Runner.class). The runner
is what is given a suite class and runs its tests (propagating
test execution events to any interested listeners). We use
RandomizedRunner which manages certain things on top of the default
JUnit runner (thread groups, custom annotation for seeds, custom test
methods in junit3 style, etc.).

2) ant's task for executing JUnit tests (junit4). This one is
responsible for collecting suites, forking jvms and managing
listeners. It is also responsible for failing ant's build (by throwing
an exception) if requested -- see "haltonfailure" property here
http://labs.carrotsearch.com/download/randomizedtesting/1.3.0/docs/junit4-a=
nt/Tasks/junit4.html.
This is consistent with ANT's default runner.

There is some confusion about the two -- in Lucene we use both but you
could run suites annotated with RandomizedRunner using any container
you want (including standard ANT's <junit> task).

Unfortunately this also means that whether a build is failed or not is
a direct consequence of if any of the tests failed or not. There are
no other conditions for this (including complex conditions you
mentioned).

> I know that things like jenkins are totally fine with the idea of a build
> succeeding even if some of the junit testsuite.xml files contain failures

This is the "haltonfailure" option. You can actually override it from
command line in Lucene build scripts and run full build without
stopping on errors -- see ant test-help:

  [echo] # Run all tests without stopping on errors (inspect log files!).
  [echo] ant -Dtests.haltonfailure=3Dfalse test

> health) but the key question is could we have our test runner say "test
> X failed, therefore the build should fail" but also "test Y failed, and
> test Y is annotated with @UnstableTest, therefore don't let that failure
> fail th entire build.

Not really. Not in a way that would be elegant and fit into JUnit
listener infrastructure. Read on.

> Ultimatley i think it's important that these failures be reported as
> faulres -- because that's truly what they are -- we shouldn't try to
> sugar coat it, or pretend something happened that didn't. =C2=A0 Ideally =
these

+1.

> I think a Version 2.0 "feature" would be to see agregated historic stats
> on the pass/fail rate of every test, regardless of it's annotation, so we
> can see:
> =C2=A0a) statistically, how often does test X fail on jenkins?
> =C2=A0b) statistically, how often does test X fail on my box?
> =C2=A0c) statistically, how often does test X fail on your box? oh really=
 -
> that's the same stats that Pete is seeing, but much higher then anyone
> else including jenkins and you both run Windows, so maybe there is a
> platform specific bug in the code and/or test?

This is an interesting idea and I think this could be done by adding a
custom report and some marker in an assumption-ignore status... The
history is doable (much like execution times currently)... but it's
hackish.

> =C2=A01) "ant test"
> =C2=A0 =C2=A0- treats @UnstableTest the same as @AwaitsFix
> =C2=A0 =C2=A0- fails the build if any test fails
> =C2=A02) "ant test-unstable"
> =C2=A0 =C2=A0- *only* runs @UnstableTest tests
> =C2=A0 =C2=A0- doesn't fail the build for any reason
> =C2=A0 =C2=A0- puts the result XML files in the same place as "ant test"
> =C2=A0 =C2=A0 =C2=A0(so jenkins UI sees them)
> =C2=A03) jenkins runs "ant test test-unstable"

This is doable by enabling/disabling test groups. A new build plan
would need to be created that would do:

ant -Dtests.haltonfailure=3Dfalse -Dtests.awaitsfix=3Dtrue
-Dtests.unstable=3Dtrue test

both awaitsfix and unstable would be disabled by default (and
assumption-ignored) so this wouldn't affect anybody else. The above
would run all tests without stopping on errors. A post-processing
script could then parse json reports (or XML reports) and collect
historical statistics.

Doable, but honestly this seems like more work (scripts for collecting
stats, test groups are trivial) than trying to fix those two or three
tests that fail?

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org