harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oliver Deakin <oliver.dea...@googlemail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Tue, 18 Jul 2006 16:01:38 GMT
George Harley wrote:
> Oliver Deakin wrote:
>> George Harley wrote:
>>> Oliver Deakin wrote:
>>>> George Harley wrote:
>>>>> <SNIP!>
>>>>>
>>>>> Here the annotation on MyTestClass applies to all of its test 
>>>>> methods.
>>>>>
>>>>> So what are the well-known TestNG groups that we could define for 
>>>>> use inside Harmony ? Here are some of my initial thoughts:
>>>>>
>>>>>
>>>>> * type.impl  --  tests that are specific to Harmony
>>>>
>>>> So tests are implicitly API unless specified otherwise?
>>>>
>>>> I'm slightly confused by your definition of impl tests as "tests 
>>>> that are
>>>> specific to Harmony". Does this mean that impl tests are only
>>>> those that test classes in org.apache.harmony packages?
>>>> I thought that impl was our way of saying "tests that need to go on
>>>> the bootclasspath".
>>>>
>>>> I think I just need a little clarification...
>>>>
>>>
>>> Hi Oliver,
>>>
>>> I was using the definition of implementation-specific tests that we 
>>> currently have on the Harmony testing conventions web page. That is, 
>>> implementation-specific tests are those that are dependent on some 
>>> aspect of the Harmony implementation and would therefore not pass 
>>> when run against the RI or other conforming implementations. It's 
>>> orthogonal to the classpath/bootclasspath issue.
>>
>> OK, that's what I imagined you meant. IMHO using api and impl
>> in this way makes the most sense (since, as you say, they do not
>> really relate to the classpath/bootclasspath issue).
>>
>> So do we also need a pair of groups for classpath/bootclasspath
>> tests? I'm assuming that this is how we would handle this distinction,
>> rather than organising them into separate directories in the file 
>> system.
>>
>
> Hi Oliver,
>
>
> I guess that would be possible but, given that the intended 
> classloader of a test is probably not going to vary much, I would have 
> no objection to that distinction being made in separate directories in 
> the file system. In other words, I see the benefits of a suite-driven 
> system (TestNG or whatever) as being more applicable to those 
> attributes of tests that we identify as being more susceptible to 
> change. Examples like "currently only works on platform X" or "broken 
> on platform Y" or "only works when run against Harmony" seem like good 
> candidates, especially when the time comes to grow Harmony on other 
> platforms. Right now, I don't see the runtime classloader fitting into 
> that category. Maybe I'm wrong.
>

Right - the only thing that prompted me to ask about it was the 
possibility of
uniting tests for a particular class into a single test class. With a 
separate
directory structure for classpath and bootclasspath tests there are 
often two
test classes for each class-under-test - making this distinction with
groups allows us to keep all tests for a class in a single file.

However, I don't feel strongly about this - as long as the distinction 
is clear
and easy to maintain then I'm happy. Also, I seem to remember a
discussion about splitting the tests up into separate classpath and
bootclasspath directories so that IDEs could compile them into
separate bin directories, and run them with the right config when not
using the Ant scripts [1]. This seems like a decent reason to go
with separate directories.

Regards,
Oliver

[1] 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c446C4328.80702@googlemail.com%3e

>
>>>
>>>
>>>>> * state.broken.<platform id>  --  tests bust on a specific platform
>>>>>
>>>>> * state.broken  --  tests broken on every platform but we want to 
>>>>> decide whether or not to run from our suite configuration
>>>>>
>>>>> * os.<platform id>  --  tests that are to be run only on the 
>>>>> specified platform (a test could be member of more than one of these)
>>>>
>>>> And the defaults for these are an unbroken state and runs on any 
>>>> platform.
>>>> That makes sense...
>>>>
>>>> Will the platform ids be organised in a similar way to the platform 
>>>> ids
>>>> we've discussed before for organisation of native code [1]?
>>>>
>>>
>>> The actual string used to identify a particular platform can be 
>>> whatever we want it to be, just so long as we are consistent. So, 
>>> yes, the ids mentioned in the referenced email would seem a good 
>>> starting point. Do we need to include a 32-bit/64-bit identifier ?
>>
>> I cannot immediately think of any obvious 32/64-bit specific tests 
>> that we
>> might require in the future (although Id be interested to know if anyone
>> can think of any!). However, if the need did arise, then I would
>> suggest that this is incorporated as another tag on the end of the
>> group name e.g. os.linux.ppc.32.
>>
>>
>
> Right.
>
>>>
>>>
>>>> So all tests are, by default, in an all-platforms (or shared) group.
>>>> If a test fails on all Windows platforms, it is marked with
>>>> state.broken.windows.
>>>> If a test fails on Windows but only on, say, amd hardware,
>>>> it is marked state.broken.windows.amd.
>>>>
>>>
>>> Yes. Agreed.
>>>
>>>
>>>> Then when you come to run tests on your windows amd machine,
>>>> you want to include all tests in the all-platform (shared) group,
>>>> os.windows and os.windows.amd, and exclude all tests in
>>>> the state.broken, state.broken.windows and state.broken.windows.amd
>>>> groups.
>>>>
>>>> Does this tally with what you were thinking?
>>>>
>>>
>>> Yes, that is the idea.
>>>
>>>
>>>>>
>>>>>
>>>>> What does everyone else think ? Does such a scheme sound reasonable ?
>>>>
>>>> I think so - it seems to cover our current requirements. Thanks for
>>>> coming up with this!
>>>>
>>>
>>> Thanks, but I don't see it as final yet really. It would be great to 
>>> prove the worth of this by doing a trial on one of the existing 
>>> modules, ideally something that contains tests that are 
>>> platform-specific.
>>
>> Thanks for volunteering... ;)
>>
>> ...but seriously, do any of our modules currently contain platform 
>> specific tests?
>> Have you attempted a TestNG trial on any of the modules (with or
>> without platform specific tests) and, if so, was it 
>> simpler/harder/better/worse
>> than our current setup?
>>
>> Regards,
>> Oliver
>>
>
> Yes, I believe that there are platform specific tests out there. NIO 
> and auth are two examples that spring to mind. My own experiments with 
> TestNG have so far not been with our modules but in separate projects. 
> Running the tests didn't seem all that different to what we currently 
> use but the speed with which the configuration could be altered was a 
> lot better, not to mention the degree of control available.
>
> Best regards,
> George
>
>
>>>
>>> Best regards,
>>> George
>>>
>>>
>>>> Regards,
>>>> Oliver
>>>>
>>>> [1] 
>>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>>>>
>>>>
>>>>>
>>>>> Thanks for reading this far.
>>>>>
>>>>> Best regards,
>>>>> George
>>>>>
>>>>>
>>>>>
>>>>> George Harley wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Just seen Tim's note on test support classes and it really caught

>>>>>> my attention as I have been mulling over this issue for a little

>>>>>> while now. I think that it is a good time for us to return to the

>>>>>> topic of class library test layouts.
>>>>>>
>>>>>> The current proposal [1] sets out to segment our different types

>>>>>> of test by placing them in different file locations. After 
>>>>>> looking at the recent changes to the LUNI module tests (where the

>>>>>> layout guidelines were applied) I have a real concern that there

>>>>>> are serious problems with this approach. We have started down a 
>>>>>> track of just continually growing the number of test source 
>>>>>> folders as new categories of test are identified and IMHO that is

>>>>>> going to bring complexity and maintenance issues with these tests.
>>>>>>
>>>>>> Consider the dimensions of tests that we have ...
>>>>>>
>>>>>> API
>>>>>> Harmony-specific
>>>>>> Platform-specific
>>>>>> Run on classpath
>>>>>> Run on bootclasspath
>>>>>> Behaves different between Harmony and RI
>>>>>> Stress
>>>>>> ...and so on...
>>>>>>
>>>>>>
>>>>>> If you weigh up all of the different possible permutations and 
>>>>>> then consider that the above list is highly likely to be extended

>>>>>> as things progress it is obvious that we are eventually heading 
>>>>>> for large amounts of related test code scattered or possibly 
>>>>>> duplicated across numerous "hard wired" source directories. How 
>>>>>> maintainable is that going to be ?
>>>>>>
>>>>>> If we want to run different tests in different configurations 
>>>>>> then IMHO we need to be thinking a whole lot smarter. We need to

>>>>>> be thinking about keeping tests for specific areas of 
>>>>>> functionality together (thus easing maintenance); we need 
>>>>>> something quick and simple to re-configure if necessary (pushing

>>>>>> whole directories of files around the place does not seem a 
>>>>>> particularly lightweight approach); and something that is not 
>>>>>> going to potentially mess up contributed patches when the file 
>>>>>> they patch is found to have been recently pushed from source 
>>>>>> folder A to B.
>>>>>>
>>>>>> To connect into another recent thread, there have been some posts

>>>>>> lately about handling some test methods that fail on Harmony and

>>>>>> have meant that entire test case classes have been excluded from

>>>>>> our test runs. I have also been noticing some API test methods 
>>>>>> that pass fine on Harmony but fail when run against the RI. Are 
>>>>>> the different behaviours down to errors in the Harmony 
>>>>>> implementation ? An error in the RI implementation ? A bug in the

>>>>>> RI Javadoc ? Only after some investigation has been carried out 
>>>>>> do we know for sure. That takes time. What do we do with the test

>>>>>> methods in the meantime ? Do we push them round the file system 
>>>>>> into yet another new source folder ? IMHO we need a testing 
>>>>>> strategy that enables such "problem" methods to be tracked easily

>>>>>> without disruption to the rest of the other tests.
>>>>>>
>>>>>> A couple of weeks ago I mentioned that the TestNG framework [2] 
>>>>>> seemed like a reasonably good way of allowing us to both group 
>>>>>> together different kinds of tests and permit the exclusion of 
>>>>>> individual tests/groups of tests [3]. I would like to strongly 
>>>>>> propose that we consider using TestNG as a means of providing the

>>>>>> different test configurations required by Harmony. Using a 
>>>>>> combination of annotations and XML to capture the kinds of 
>>>>>> sophisticated test configurations that people need, and that 
>>>>>> allows us to specify down to the individual method, has got to be

>>>>>> more scalable and flexible than where we are headed now.
>>>>>>
>>>>>> Thanks for reading this far.
>>>>>>
>>>>>> Best regards,
>>>>>> George
>>>>>>
>>>>>>
>>>>>> [1] 
>>>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html

>>>>>>
>>>>>> [2] http://testng.org
>>>>>> [3] 
>>>>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: 
>>>>> harmony-dev-help@incubator.apache.org
>>>>>
>>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message