harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hindess <mark.hind...@googlemail.com>
Subject Re: test excludes
Date Fri, 28 May 2010 13:57:32 GMT

In message <AANLkTiknGv4g0UIQlMHp2D7hCd71l7mXNQoelniZgsao@mail.gmail.com>,
Nathan Beyer writes:
>
> On Thu, May 27, 2010 at 9:12 AM, Mark Hindess
> <mark.hindess@googlemail.com> wrote:
> >
> > In message <AANLkTikEMRMbIYQVA2FnpKr4VCvAW7AF6LWmIG8Ns8S7@mail.gmail.com>,
> > Nathan Beyer writes:
> >>
> >> On Wed, May 26, 2010 at 4:59 AM, Mark Hindess
> >> <mark.hindess@googlemail.com> wrote:
> >> >
> >> >
> >> > Last night I decided to try to implement this simple exclude list
> >> > mechanism in my build-improvement branch.  I've checked in my
> >> > changes at
> >> > https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/mrh@948377
> >>
> >> I don't want to be rude,
> >
> > Not at all; I don't think you've ever been rude on this list.
> >
> >> but ... I brought this up a few months back and the response I
> >> got was less than overwhelming. I stopped my work.  How is this
> >> different?
> >>
> >> http://harmony.markmail.org/message/4rhomwqjdr5uklln?q=3Djunit+annotation
> >
> > I think the response you got was fairly positive.  The only issue
> > appears to be a request from Jesse for some way to externally
> > influence the test selection process which I don't think was
> > tackled.
> >
> > How far did you get?  I'd be interested in playing with the
> > annotations if you've written them.  I'm not convinced annotations
> > are enough but they probably will be part of the right longer term
> > solution.
> 
> What I have now is -
> * Platform - an annotation that can mark a class and/or method with a
>   os, arch and vm value
> * PlatformRule - a org.junit.rules.MethodRule implementation, which
> is instantiated and stored as an instance variable in unit tests that
> use the Platform annotation; when a test with this rule is run, if the
> class or method is annotated with Platform, only the test methods that
> match all qualifications will be run; methods not run will be logged
> to standard error.

It would be nice to be able to do things like:

  @Exclude()          // for exclude.{common,intern} tests
  @Exclude(os=linux)  // for tests that fail on any linux platform

but to start with so long as we can have multiple @Exclude annotation
this wont be an urgent problem to solve.

> This was my experiment for using annotations and method rules. The
> next step would be to define some annotations for the test
> qualifications we wanted to express - Exclude(os, arch, vm), etc. And
> then write method rules that would massage the run based on these
> rules.

I assume the Exclude annotation wont be a big leap from the Platform
annotation.  If you can provide this I can use a similar Perl one-liner
to the one I used for the Support_Excludes changes in my branch to apply
them to all methods in classes in the current exclude lists.  We can
then refine them after checking which methods really need to be excluded
as we get time to go through them.

> The interesting thing about method rules is that they can affect
> the test runs without requiring any special runner, however, when a
> method rule is used to not run a method, most runners will treat the
> methods as run, even if the method rule doesn't actually execute the
> 'statement'.

Method rules would be good enough for now but having the top-level
junit report showing more tests run even though they might have tested
nothing is not ideal.  So, I think longer term a runner will be
necessary/useful.  (I'd expect Jesse would override the runner with his
own to get whatever external filtering he needs.)

Regards,
 Mark.



Mime
View raw message