harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Wed, 19 Jul 2006 08:48:39 GMT
According to "TestNG Ant Task" [1], it seems that the TestNG Ant task 
does not support to fork a new JVM, that is, we must launch ant using 
Harmony itself. Any comments? Thanks a lot.

[1]http://testng.org/doc/ant.html

Best regards,
Richard

George Harley wrote:
> Andrew Zhang wrote:
>> On 7/18/06, George Harley <george.c.harley@googlemail.com> wrote:
>>>
>>> Oliver Deakin wrote:
>>> > George Harley wrote:
>>> >> <SNIP!>
>>> >>
>>> >> Here the annotation on MyTestClass applies to all of its test 
>>> methods.
>>> >>
>>> >> So what are the well-known TestNG groups that we could define for 
>>> use
>>> >> inside Harmony ? Here are some of my initial thoughts:
>>> >>
>>> >>
>>> >> * type.impl  --  tests that are specific to Harmony
>>> >
>>> > So tests are implicitly API unless specified otherwise?
>>> >
>>> > I'm slightly confused by your definition of impl tests as "tests that
>>> are
>>> > specific to Harmony". Does this mean that impl tests are only
>>> > those that test classes in org.apache.harmony packages?
>>> > I thought that impl was our way of saying "tests that need to go on
>>> > the bootclasspath".
>>> >
>>> > I think I just need a little clarification...
>>> >
>>>
>>> Hi Oliver,
>>>
>>> I was using the definition of implementation-specific tests that we
>>> currently have on the Harmony testing conventions web page. That is,
>>> implementation-specific tests are those that are dependent on some
>>> aspect of the Harmony implementation and would therefore not pass when
>>> run against the RI or other conforming implementations. It's orthogonal
>>> to the classpath/bootclasspath issue.
>>>
>>>
>>> >> * state.broken.<platform id>  --  tests bust on a specific platform
>>> >>
>>> >> * state.broken  --  tests broken on every platform but we want to
>>> >> decide whether or not to run from our suite configuration
>>> >>
>>> >> * os.<platform id>  --  tests that are to be run only on the
>>> >> specified platform (a test could be member of more than one of 
>>> these)
>>> >
>>> > And the defaults for these are an unbroken state and runs on any
>>> > platform.
>>> > That makes sense...
>>> >
>>> > Will the platform ids be organised in a similar way to the 
>>> platform ids
>>> > we've discussed before for organisation of native code [1]?
>>> >
>>>
>>> The actual string used to identify a particular platform can be 
>>> whatever
>>> we want it to be, just so long as we are consistent. So, yes, the ids
>>> mentioned in the referenced email would seem a good starting point. Do
>>> we need to include a 32-bit/64-bit identifier ?
>>>
>>>
>>> > So all tests are, by default, in an all-platforms (or shared) group.
>>> > If a test fails on all Windows platforms, it is marked with
>>> > state.broken.windows.
>>> > If a test fails on Windows but only on, say, amd hardware,
>>> > it is marked state.broken.windows.amd.
>>> >
>>>
>>> Yes. Agreed.
>>>
>>>
>>> > Then when you come to run tests on your windows amd machine,
>>> > you want to include all tests in the all-platform (shared) group,
>>> > os.windows and os.windows.amd, and exclude all tests in
>>> > the state.broken, state.broken.windows and state.broken.windows.amd
>>> > groups.
>>> >
>>> > Does this tally with what you were thinking?
>>> >
>>>
>>> Yes, that is the idea.
>>>
>>>
>>> >>
>>> >>
>>> >> What does everyone else think ? Does such a scheme sound 
>>> reasonable ?
>>> >
>>> > I think so - it seems to cover our current requirements. Thanks for
>>> > coming up with this!
>>> >
>>>
>>> Thanks, but I don't see it as final yet really. It would be great to
>>> prove the worth of this by doing a trial on one of the existing 
>>> modules,
>>> ideally something that contains tests that are platform-specific.
>>
>>
>> Hello George, how about doing a trial on NIO module?
>>
>> So far as I know, there are several platform dependent tests in NIO 
>> module.
>> :)
>>
>> The assert statements are commented out in these tests, with "FIXME" 
>> mark.
>>
>> Furthurmore, I also find some platform dependent behaviours of 
>> FileChannel.
>> If TestNG is applied on NIO, I will supplement new tests for 
>> FileChannel and
>> fix the bug of source code.
>>
>> What's your opnion? Any suggestions/comments?
>>
>> Thanks!
>>
>
> Hi Andrew,
>
> That sounds like a very good idea. If there is agreement in the 
> project that 5.0 annotations are the way to go (as opposed to the 
> pre-5.0 Javadoc comment support offered by TestNG) then to the best of 
> my knowledge all that is stopping us from doing this trial is the lack 
> of a 5.0 VM to run the Harmony tests on. Hopefully that will be 
> addressed soon. When it is I would be happy to get stuck into this trial.
>
> Best regards,
> George
>
>
>> Best regards,
>>> George
>>>
>>>
>>> > Regards,
>>> > Oliver
>>> >
>>> > [1]
>>> >
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>>>
>>> >
>>> >
>>> >>
>>> >> Thanks for reading this far.
>>> >>
>>> >> Best regards,
>>> >> George
>>> >>
>>> >>
>>> >>
>>> >> George Harley wrote:
>>> >>> Hi,
>>> >>>
>>> >>> Just seen Tim's note on test support classes and it really 
>>> caught my
>>> >>> attention as I have been mulling over this issue for a little while
>>> >>> now. I think that it is a good time for us to return to the 
>>> topic of
>>> >>> class library test layouts.
>>> >>>
>>> >>> The current proposal [1] sets out to segment our different types
of
>>> >>> test by placing them in different file locations. After looking
at
>>> >>> the recent changes to the LUNI module tests (where the layout
>>> >>> guidelines were applied) I have a real concern that there are
>>> >>> serious problems with this approach. We have started down a 
>>> track of
>>> >>> just continually growing the number of test source folders as new
>>> >>> categories of test are identified and IMHO that is going to bring
>>> >>> complexity and maintenance issues with these tests.
>>> >>>
>>> >>> Consider the dimensions of tests that we have ...
>>> >>>
>>> >>> API
>>> >>> Harmony-specific
>>> >>> Platform-specific
>>> >>> Run on classpath
>>> >>> Run on bootclasspath
>>> >>> Behaves different between Harmony and RI
>>> >>> Stress
>>> >>> ...and so on...
>>> >>>
>>> >>>
>>> >>> If you weigh up all of the different possible permutations and then
>>> >>> consider that the above list is highly likely to be extended as
>>> >>> things progress it is obvious that we are eventually heading for
>>> >>> large amounts of related test code scattered or possibly duplicated
>>> >>> across numerous "hard wired" source directories. How 
>>> maintainable is
>>> >>> that going to be ?
>>> >>>
>>> >>> If we want to run different tests in different configurations then
>>> >>> IMHO we need to be thinking a whole lot smarter. We need to be
>>> >>> thinking about keeping tests for specific areas of functionality
>>> >>> together (thus easing maintenance); we need something quick and
>>> >>> simple to re-configure if necessary (pushing whole directories of
>>> >>> files around the place does not seem a particularly lightweight
>>> >>> approach); and something that is not going to potentially mess up
>>> >>> contributed patches when the file they patch is found to have been
>>> >>> recently pushed from source folder A to B.
>>> >>>
>>> >>> To connect into another recent thread, there have been some posts
>>> >>> lately about handling some test methods that fail on Harmony and
>>> >>> have meant that entire test case classes have been excluded from

>>> our
>>> >>> test runs. I have also been noticing some API test methods that

>>> pass
>>> >>> fine on Harmony but fail when run against the RI. Are the different
>>> >>> behaviours down to errors in the Harmony implementation ? An error
>>> >>> in the RI implementation ? A bug in the RI Javadoc ? Only after

>>> some
>>> >>> investigation has been carried out do we know for sure. That takes
>>> >>> time. What do we do with the test methods in the meantime ? Do we
>>> >>> push them round the file system into yet another new source 
>>> folder ?
>>> >>> IMHO we need a testing strategy that enables such "problem" methods
>>> >>> to be tracked easily without disruption to the rest of the other
>>> tests.
>>> >>>
>>> >>> A couple of weeks ago I mentioned that the TestNG framework [2]
>>> >>> seemed like a reasonably good way of allowing us to both group
>>> >>> together different kinds of tests and permit the exclusion of
>>> >>> individual tests/groups of tests [3]. I would like to strongly
>>> >>> propose that we consider using TestNG as a means of providing the
>>> >>> different test configurations required by Harmony. Using a
>>> >>> combination of annotations and XML to capture the kinds of
>>> >>> sophisticated test configurations that people need, and that allows
>>> >>> us to specify down to the individual method, has got to be more
>>> >>> scalable and flexible than where we are headed now.
>>> >>>
>>> >>> Thanks for reading this far.
>>> >>>
>>> >>> Best regards,
>>> >>> George
>>> >>>
>>> >>>
>>> >>> [1]
>>> >>>
>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>>
>>> >>>
>>> >>> [2] http://testng.org
>>> >>> [3]
>>> >>>
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >> 
>>> ---------------------------------------------------------------------
>>> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> >> For additional commands, e-mail: 
>>> harmony-dev-help@incubator.apache.org
>>> >>
>>> >>
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message