Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of oliver.deakin@googlemail.com
 designates 64.233.182.190 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=googlemail.com;
        h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
        b=BME4+XXgTSBtg3lGgMGPVNiOAs67+1ne5uMfvMKeFAikr2fhR9pN6Bgzj94dFAjB8+Hlm68n4tCXlHHb8cnXErJsiQdRG3OOxYqQJctCHgsaJiZdLiUgdGy95HAiU//Vm9FsTTUUzY9FBH++yd9WlwW2RX7aUHUm1bsr4Wh2Kxc=
Message-ID: <44BCB1F6.3020208@googlemail.com>
Date: Tue, 18 Jul 2006 11:03:34 +0100
From: Oliver Deakin <oliver.deakin@googlemail.com>
User-Agent: Thunderbird 1.5.0.4 (Windows/20060516)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib] Testing conventions - a proposal
References: <44ABB451.30806@googlemail.com> <44B75E41.7020901@googlemail.com>
In-Reply-To: <44B75E41.7020901@googlemail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

George Harley wrote:
> <SNIP!>
>
> Here the annotation on MyTestClass applies to all of its test methods.
>
> So what are the well-known TestNG groups that we could define for use 
> inside Harmony ? Here are some of my initial thoughts:
>
>
> * type.impl  --  tests that are specific to Harmony

So tests are implicitly API unless specified otherwise?

I'm slightly confused by your definition of impl tests as "tests that are
specific to Harmony". Does this mean that impl tests are only
those that test classes in org.apache.harmony packages?
I thought that impl was our way of saying "tests that need to go on
the bootclasspath".

I think I just need a little clarification...

> * state.broken.<platform id>  --  tests bust on a specific platform
>
> * state.broken  --  tests broken on every platform but we want to 
> decide whether or not to run from our suite configuration
>
> * os.<platform id>  --  tests that are to be run only on the specified 
> platform (a test could be member of more than one of these)

And the defaults for these are an unbroken state and runs on any platform.
That makes sense...

Will the platform ids be organised in a similar way to the platform ids
we've discussed before for organisation of native code [1]?

So all tests are, by default, in an all-platforms (or shared) group.
If a test fails on all Windows platforms, it is marked with
state.broken.windows.
If a test fails on Windows but only on, say, amd hardware,
it is marked state.broken.windows.amd.

Then when you come to run tests on your windows amd machine,
you want to include all tests in the all-platform (shared) group,
os.windows and os.windows.amd, and exclude all tests in
the state.broken, state.broken.windows and state.broken.windows.amd
groups.

Does this tally with what you were thinking?

>
>
> What does everyone else think ? Does such a scheme sound reasonable ?

I think so - it seems to cover our current requirements. Thanks for
coming up with this!

Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>
> Thanks for reading this far.
>
> Best regards,
> George
>
>
>
> George Harley wrote:
>> Hi,
>>
>> Just seen Tim's note on test support classes and it really caught my 
>> attention as I have been mulling over this issue for a little while 
>> now. I think that it is a good time for us to return to the topic of 
>> class library test layouts.
>>
>> The current proposal [1] sets out to segment our different types of 
>> test by placing them in different file locations. After looking at 
>> the recent changes to the LUNI module tests (where the layout 
>> guidelines were applied) I have a real concern that there are serious 
>> problems with this approach. We have started down a track of just 
>> continually growing the number of test source folders as new 
>> categories of test are identified and IMHO that is going to bring 
>> complexity and maintenance issues with these tests.
>>
>> Consider the dimensions of tests that we have ...
>>
>> API
>> Harmony-specific
>> Platform-specific
>> Run on classpath
>> Run on bootclasspath
>> Behaves different between Harmony and RI
>> Stress
>> ...and so on...
>>
>>
>> If you weigh up all of the different possible permutations and then 
>> consider that the above list is highly likely to be extended as 
>> things progress it is obvious that we are eventually heading for 
>> large amounts of related test code scattered or possibly duplicated 
>> across numerous "hard wired" source directories. How maintainable is 
>> that going to be ?
>>
>> If we want to run different tests in different configurations then 
>> IMHO we need to be thinking a whole lot smarter. We need to be 
>> thinking about keeping tests for specific areas of functionality 
>> together (thus easing maintenance); we need something quick and 
>> simple to re-configure if necessary (pushing whole directories of 
>> files around the place does not seem a particularly lightweight 
>> approach); and something that is not going to potentially mess up 
>> contributed patches when the file they patch is found to have been 
>> recently pushed from source folder A to B.
>>
>> To connect into another recent thread, there have been some posts 
>> lately about handling some test methods that fail on Harmony and have 
>> meant that entire test case classes have been excluded from our test 
>> runs. I have also been noticing some API test methods that pass fine 
>> on Harmony but fail when run against the RI. Are the different 
>> behaviours down to errors in the Harmony implementation ? An error in 
>> the RI implementation ? A bug in the RI Javadoc ? Only after some 
>> investigation has been carried out do we know for sure. That takes 
>> time. What do we do with the test methods in the meantime ? Do we 
>> push them round the file system into yet another new source folder ? 
>> IMHO we need a testing strategy that enables such "problem" methods 
>> to be tracked easily without disruption to the rest of the other tests.
>>
>> A couple of weeks ago I mentioned that the TestNG framework [2] 
>> seemed like a reasonably good way of allowing us to both group 
>> together different kinds of tests and permit the exclusion of 
>> individual tests/groups of tests [3]. I would like to strongly 
>> propose that we consider using TestNG as a means of providing the 
>> different test configurations required by Harmony. Using a 
>> combination of annotations and XML to capture the kinds of 
>> sophisticated test configurations that people need, and that allows 
>> us to specify down to the individual method, has got to be more 
>> scalable and flexible than where we are headed now.
>>
>> Thanks for reading this far.
>>
>> Best regards,
>> George
>>
>>
>> [1] 
>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>
>> [2] http://testng.org
>> [3] 
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e 
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org