Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (asf.osuosl.org: 202.81.18.155 is neither permitted nor
 denied by domain of richard.liangyx@gmail.com)
Message-ID: <44AC6DF5.30100@gmail.com>
Date: Thu, 06 Jul 2006 09:57:09 +0800
From: Richard Liang <richard.liangyx@gmail.com>
User-Agent: Thunderbird 1.5.0.4 (Windows/20060516)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib] Testing conventions - a proposal
References: <44ABB451.30806@googlemail.com>
In-Reply-To: <44ABB451.30806@googlemail.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=ISO-8859-1; format=flowed


George Harley wrote:
> Hi,
>
> Just seen Tim's note on test support classes and it really caught my 
> attention as I have been mulling over this issue for a little while 
> now. I think that it is a good time for us to return to the topic of 
> class library test layouts.
>
> The current proposal [1] sets out to segment our different types of 
> test by placing them in different file locations. After looking at the 
> recent changes to the LUNI module tests (where the layout guidelines 
> were applied) I have a real concern that there are serious problems 
> with this approach. We have started down a track of just continually 
> growing the number of test source folders as new categories of test 
> are identified and IMHO that is going to bring complexity and 
> maintenance issues with these tests.
Yes, you'll see our ant scripts get more and more complex. :-)

>
> Consider the dimensions of tests that we have ...
>
> API
> Harmony-specific
> Platform-specific
> Run on classpath
> Run on bootclasspath
> Behaves different between Harmony and RI
> Stress
> ...and so on...
>
>
> If you weigh up all of the different possible permutations and then 
> consider that the above list is highly likely to be extended as things 
> progress it is obvious that we are eventually heading for large 
> amounts of related test code scattered or possibly duplicated across 
> numerous "hard wired" source directories. How maintainable is that 
> going to be ?
>
> If we want to run different tests in different configurations then 
> IMHO we need to be thinking a whole lot smarter. We need to be 
> thinking about keeping tests for specific areas of functionality 
> together (thus easing maintenance); we need something quick and simple 
> to re-configure if necessary (pushing whole directories of files 
> around the place does not seem a particularly lightweight approach); 
> and something that is not going to potentially mess up contributed 
> patches when the file they patch is found to have been recently pushed 
> from source folder A to B.
>
> To connect into another recent thread, there have been some posts 
> lately about handling some test methods that fail on Harmony and have 
> meant that entire test case classes have been excluded from our test 
> runs. I have also been noticing some API test methods that pass fine 
> on Harmony but fail when run against the RI. Are the different 
> behaviours down to errors in the Harmony implementation ? An error in 
> the RI implementation ? A bug in the RI Javadoc ? Only after some 
> investigation has been carried out do we know for sure. That takes 
> time. What do we do with the test methods in the meantime ? Do we push 
> them round the file system into yet another new source folder ? IMHO 
> we need a testing strategy that enables such "problem" methods to be 
> tracked easily without disruption to the rest of the other tests.
It's really worth thinking about our testing strategy...
>
> A couple of weeks ago I mentioned that the TestNG framework [2] seemed 
> like a reasonably good way of allowing us to both group together 
> different kinds of tests and permit the exclusion of individual 
> tests/groups of tests [3]. I would like to strongly propose that we 
> consider using TestNG as a means of providing the different test 
> configurations required by Harmony. Using a combination of annotations 
> and XML to capture the kinds of sophisticated test configurations that 
> people need, and that allows us to specify down to the individual 
> method, has got to be more scalable and flexible than where we are 
> headed now.
>
Will try to study TestNG before I can give comment ;-)

> Thanks for reading this far.
>
> Best regards,
> George
>
>
> [1] 
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>
> [2] http://testng.org
> [3] 
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e 
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Richard Liang
China Software Development Lab, IBM 


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org