harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Beyer <ndbe...@apache.org>
Subject Re: test excludes
Date Fri, 28 May 2010 02:02:36 GMT
On Thu, May 27, 2010 at 9:12 AM, Mark Hindess
<mark.hindess@googlemail.com> wrote:
>
> In message <AANLkTikEMRMbIYQVA2FnpKr4VCvAW7AF6LWmIG8Ns8S7@mail.gmail.com>,
> Nathan Beyer writes:
>>
>> On Wed, May 26, 2010 at 4:59 AM, Mark Hindess
>> <mark.hindess@googlemail.com> wrote:
>> >
>> >
>> > Last night I decided to try to implement this simple exclude list
>> > mechanism in my build-improvement branch.  I've checked in my changes at
>> > https://svn.apache.org/repos/asf/harmony/enhanced/java/branches/mrh@948377
>>
>> I don't want to be rude,
>
> Not at all; I don't think you've ever been rude on this list.
>
>> but ... I brought this up a few months back and the response I got was
>> less than overwhelming. I stopped my work.
>> How is this different?
>>
>> http://harmony.markmail.org/message/4rhomwqjdr5uklln?q=junit+annotation
>
> I think the response you got was fairly positive.  The only issue
> appears to be a request from Jesse for some way to externally influence
> the test selection process which I don't think was tackled.
>
> How far did you get?  I'd be interested in playing with the annotations
> if you've written them.  I'm not convinced annotations are enough but
> they probably will be part of the right longer term solution.

What I have now is -
* Platform - an annotation that can mark a class and/or method with a
os, arch and vm value
* PlatformRule - a org.junit.rules.MethodRule implementation, which is
instantiated and stored as an instance variable in unit tests that use
the Platform annotation; when a test with this rule is run, if the
class or method is annotated with Platform, only the test methods that
match all qualifications will be run; methods not run will be logged
to standard error.

This was my experiment for using annotations and method rules. The
next step would be to define some annotations for the test
qualifications we wanted to express - Exclude(os, arch, vm), etc. And
then write method rules that would massage the run based on these
rules.

The interesting thing about method rules is that they can affect the
test runs without requiring any special runner, however, when a method
rule is used to not run a method, most runners will treat the methods
as run, even if the method rule doesn't actually execute the
'statement'.

-Nathan

>
> I don't see what I have done as a long term solution - and tried to make
> it clear that I didn't.  My motivation has been documented elsewhere in
> this thread - particularly, in my reply to Tim's equally blunt
> response! ;-)  I also don't see how what I have proposed doing in the
> short term ... in particular ...
>
>> > [snip]
>> >
>> > I'm tempted to do away with the isExcluded() checks/lists are replace
>> > them with more readable:
>> >
>> >   if (Support_Excludes.isRI()) {
>> >
>> > or:
>> >
>> >   if (Support_Excludes.isLinux() && Support_Excludes.is64bit()) {
>> >
>> > etc.  To make the Excludes more self-contained but that requires
>> > looking at the excluded tests in more detail to understand the
>> > requirements better and as I said I wanted to make this a simple
>> > step from what we had today.
>
> ... prevents doing something better when a solution is decided upon.
> In fact, if the tests are modified like this it will make writing the
> correct annotations a lot quicker.
>
>> > I'd like to merge this to trunk/java6 after the code freeze but I'd
>> > like some feedback first.  I don't necessarily see this as a final
>> > solution but I just wanted to do something since this topic has been
>> > discussed for far to long and I wanted something that we could move
>> > to from where we are today without to much effort.  On the other
>> > hand, I'm not convinced that annotations are a good solution either
>> > since they don't give you fine-grained control so every distinction
>> > has to be represented by a separate method.
>
> In my response to Sebb, I mentioned a specific example of a single
> assert which fails on the RI.  I'd like to think that we can find a
> better solution to this example than duplicating the method.
>
> In short, when you have some annotations I can use then I will certainly
> use them to mark up the tests.  But I am impatient to get more tests
> running right now so I would like to continue to do something about that
> in the meantime.
>
> I've not done the merge from my branch anyway since it turns out I can
> get more tests running just by re-testing and un-excluding whole test so
> I am concentrating on that first.  By the time I've done that perhaps
> your annotations will be ready and I can use those in trunk instead?
>
> Regards,
>  Mark.
>
>
>

Mime
View raw message