Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (nike.apache.org: domain of nbeyer@gmail.com designates
 74.125.82.49 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=NyiPufSEC5VxEIQOUeHg5iQm927eOUlKKyN8FmKKS0Jx3P6m6mkzjXPKgW4D8CpOn5
         KrEx6oEXz4+HsxrnsbWcVf91Uj4Tx0sUVNK07W/xYHxSCIsb2R571VrfgroZAth+miuJ
         ELV3RiQYmuGZgoeMZDysbukHe7FWaPTM7zDo8=
MIME-Version: 1.0
Sender: nbeyer@gmail.com
In-Reply-To: <201005281357.o4SDvWSc003177@d12av03.megacenter.de.ibm.com>
References: <AANLkTikh3nV56QB8HXSXfzLnI9jrtU-C2cRF1shElZVt@mail.gmail.com>
	<201005250742.o4P7g1wt019461@d06av01.portsmouth.uk.ibm.com>
	<201005260959.o4Q9xI9K005150@d06av02.portsmouth.uk.ibm.com>
	<AANLkTikEMRMbIYQVA2FnpKr4VCvAW7AF6LWmIG8Ns8S7@mail.gmail.com>
	<201005271412.o4RECYHn020477@d12av02.megacenter.de.ibm.com>
	<AANLkTiknGv4g0UIQlMHp2D7hCd71l7mXNQoelniZgsao@mail.gmail.com>
	<201005281357.o4SDvWSc003177@d12av03.megacenter.de.ibm.com>
Date: Sun, 30 May 2010 18:54:27 -0500
Message-ID: <AANLkTik96jKPompjktRnDC2V70qAsCJ9DE9HiuHZQHHS@mail.gmail.com>
Subject: Re: test excludes
From: Nathan Beyer <ndbeyer@apache.org>
To: dev@harmony.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Fri, May 28, 2010 at 8:57 AM, Mark Hindess
<mark.hindess@googlemail.com> wrote:
>
> In message <AANLkTiknGv4g0UIQlMHp2D7hCd71l7mXNQoelniZgsao@mail.gmail.com>=
,
> Nathan Beyer writes:
>>
>> On Thu, May 27, 2010 at 9:12 AM, Mark Hindess
>> <mark.hindess@googlemail.com> wrote:
>> >
>> > In message <AANLkTikEMRMbIYQVA2FnpKr4VCvAW7AF6LWmIG8Ns8S7@mail.gmail.c=
om>,
>> > Nathan Beyer writes:
>> >>
>> >> On Wed, May 26, 2010 at 4:59 AM, Mark Hindess
>> >> <mark.hindess@googlemail.com> wrote:
>> >> >
>> >> >
>> >> > Last night I decided to try to implement this simple exclude list
>> >> > mechanism in my build-improvement branch. =C2=A0I've checked in my
>> >> > changes at
>> >> > https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/mrh=
@948377
>> >>
>> >> I don't want to be rude,
>> >
>> > Not at all; I don't think you've ever been rude on this list.
>> >
>> >> but ... I brought this up a few months back and the response I
>> >> got was less than overwhelming. I stopped my work. =C2=A0How is this
>> >> different?
>> >>
>> >> http://harmony.markmail.org/message/4rhomwqjdr5uklln?q=3D3Djunit+anno=
tation
>> >
>> > I think the response you got was fairly positive. =C2=A0The only issue
>> > appears to be a request from Jesse for some way to externally
>> > influence the test selection process which I don't think was
>> > tackled.
>> >
>> > How far did you get? =C2=A0I'd be interested in playing with the
>> > annotations if you've written them. =C2=A0I'm not convinced annotation=
s
>> > are enough but they probably will be part of the right longer term
>> > solution.
>>
>> What I have now is -
>> * Platform - an annotation that can mark a class and/or method with a
>> =C2=A0 os, arch and vm value
>> * PlatformRule - a org.junit.rules.MethodRule implementation, which
>> is instantiated and stored as an instance variable in unit tests that
>> use the Platform annotation; when a test with this rule is run, if the
>> class or method is annotated with Platform, only the test methods that
>> match all qualifications will be run; methods not run will be logged
>> to standard error.
>
> It would be nice to be able to do things like:
>
> =C2=A0@Exclude() =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0// for exclude.{common=
,intern} tests
> =C2=A0@Exclude(os=3Dlinux) =C2=A0// for tests that fail on any linux plat=
form
>
> but to start with so long as we can have multiple @Exclude annotation
> this wont be an urgent problem to solve.
>
>> This was my experiment for using annotations and method rules. The
>> next step would be to define some annotations for the test
>> qualifications we wanted to express - Exclude(os, arch, vm), etc. And
>> then write method rules that would massage the run based on these
>> rules.
>
> I assume the Exclude annotation wont be a big leap from the Platform
> annotation. =C2=A0If you can provide this I can use a similar Perl one-li=
ner
> to the one I used for the Support_Excludes changes in my branch to apply
> them to all methods in classes in the current exclude lists. =C2=A0We can
> then refine them after checking which methods really need to be excluded
> as we get time to go through them.
>
>> The interesting thing about method rules is that they can affect
>> the test runs without requiring any special runner, however, when a
>> method rule is used to not run a method, most runners will treat the
>> methods as run, even if the method rule doesn't actually execute the
>> 'statement'.
>
> Method rules would be good enough for now but having the top-level
> junit report showing more tests run even though they might have tested
> nothing is not ideal. =C2=A0So, I think longer term a runner will be
> necessary/useful. =C2=A0(I'd expect Jesse would override the runner with =
his
> own to get whatever external filtering he needs.)

Here's the issue -- the JUnit Ant Task doesn't provide much access to
the flexibility in JUnit4's APIs. Anything that gets into the realm of
a custom runner or customizing the run (filtering, etc) would require
a custom Ant task. If we're beholden to using the standard task, then
we're limited in what can be done. I think defining custom, semantic
annotations for the tests is the best approach for Harmony (both for
excludes, but also for includes -- i.e. run test for specific
platform), but to get them working we'll have to write, at a minimum,
a custom Ant task for running the tests.

-Nathan

>
> Regards,
> =C2=A0Mark.
>
>
>