harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Beyer <ndbe...@apache.org>
Subject Re: test excludes
Date Sun, 30 May 2010 23:54:27 GMT
On Fri, May 28, 2010 at 8:57 AM, Mark Hindess
<mark.hindess@googlemail.com> wrote:
> In message <AANLkTiknGv4g0UIQlMHp2D7hCd71l7mXNQoelniZgsao@mail.gmail.com>,
> Nathan Beyer writes:
>> On Thu, May 27, 2010 at 9:12 AM, Mark Hindess
>> <mark.hindess@googlemail.com> wrote:
>> >
>> > In message <AANLkTikEMRMbIYQVA2FnpKr4VCvAW7AF6LWmIG8Ns8S7@mail.gmail.com>,
>> > Nathan Beyer writes:
>> >>
>> >> On Wed, May 26, 2010 at 4:59 AM, Mark Hindess
>> >> <mark.hindess@googlemail.com> wrote:
>> >> >
>> >> >
>> >> > Last night I decided to try to implement this simple exclude list
>> >> > mechanism in my build-improvement branch.  I've checked in my
>> >> > changes at
>> >> > https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/mrh@948377
>> >>
>> >> I don't want to be rude,
>> >
>> > Not at all; I don't think you've ever been rude on this list.
>> >
>> >> but ... I brought this up a few months back and the response I
>> >> got was less than overwhelming. I stopped my work.  How is this
>> >> different?
>> >>
>> >> http://harmony.markmail.org/message/4rhomwqjdr5uklln?q=3Djunit+annotation
>> >
>> > I think the response you got was fairly positive.  The only issue
>> > appears to be a request from Jesse for some way to externally
>> > influence the test selection process which I don't think was
>> > tackled.
>> >
>> > How far did you get?  I'd be interested in playing with the
>> > annotations if you've written them.  I'm not convinced annotations
>> > are enough but they probably will be part of the right longer term
>> > solution.
>> What I have now is -
>> * Platform - an annotation that can mark a class and/or method with a
>>   os, arch and vm value
>> * PlatformRule - a org.junit.rules.MethodRule implementation, which
>> is instantiated and stored as an instance variable in unit tests that
>> use the Platform annotation; when a test with this rule is run, if the
>> class or method is annotated with Platform, only the test methods that
>> match all qualifications will be run; methods not run will be logged
>> to standard error.
> It would be nice to be able to do things like:
>  @Exclude()          // for exclude.{common,intern} tests
>  @Exclude(os=linux)  // for tests that fail on any linux platform
> but to start with so long as we can have multiple @Exclude annotation
> this wont be an urgent problem to solve.
>> This was my experiment for using annotations and method rules. The
>> next step would be to define some annotations for the test
>> qualifications we wanted to express - Exclude(os, arch, vm), etc. And
>> then write method rules that would massage the run based on these
>> rules.
> I assume the Exclude annotation wont be a big leap from the Platform
> annotation.  If you can provide this I can use a similar Perl one-liner
> to the one I used for the Support_Excludes changes in my branch to apply
> them to all methods in classes in the current exclude lists.  We can
> then refine them after checking which methods really need to be excluded
> as we get time to go through them.
>> The interesting thing about method rules is that they can affect
>> the test runs without requiring any special runner, however, when a
>> method rule is used to not run a method, most runners will treat the
>> methods as run, even if the method rule doesn't actually execute the
>> 'statement'.
> Method rules would be good enough for now but having the top-level
> junit report showing more tests run even though they might have tested
> nothing is not ideal.  So, I think longer term a runner will be
> necessary/useful.  (I'd expect Jesse would override the runner with his
> own to get whatever external filtering he needs.)

Here's the issue -- the JUnit Ant Task doesn't provide much access to
the flexibility in JUnit4's APIs. Anything that gets into the realm of
a custom runner or customizing the run (filtering, etc) would require
a custom Ant task. If we're beholden to using the standard task, then
we're limited in what can be done. I think defining custom, semantic
annotations for the tests is the best approach for Harmony (both for
excludes, but also for includes -- i.e. run test for specific
platform), but to get them working we'll have to write, at a minimum,
a custom Ant task for running the tests.


> Regards,
>  Mark.

View raw message