harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stepan Mishura" <stepan.mish...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Tue, 25 Jul 2006 12:47:42 GMT
On 7/20/06, George Harley  wrote:
>
> <SNIP!>
> Anyway, the point I guess that I am trying to make here is that it is
> possible in TestNG to select the methods to test dynamically using a
> little bit of scripting that (a) gives us a lot more power than the
> include/exclude technique and (b) will work the same across every
> platform we test on. Because BeanShell allows us to instantiate and use
> Java objects of any type on the classpath then the possibility of using
> more than just group membership to decide on tests to run becomes
> available to us. Please refer to the TestNG documentation for more on
> the capabilities of BeanShell and the TestNG API. I had never heard of
> it before never mind used it but still managed to get stuff working in a
> relatively short space of time.
>
> I hope this helps. Maybe I need to write a page on the wiki or something ?


Hi George,

It would be great to have your proposal for using TestNG on web-site like we
have for "Testing conventions" [1].

Thanks,
Stepan.

[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html

Best regards,
> George
>
>
>
> >> Best regards,
> >> George
> >>
> >>
> >>
> >>>> Thanks for reading this far.
> >>>>
> >>>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>>
> >>>> George Harley wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Just seen Tim's note on test support classes and it really caught
> >>>>> my attention as I have been mulling over this issue for a little
> >>>>> while now. I think that it is a good time for us to return to the
> >>>>> topic of class library test layouts.
> >>>>>
> >>>>> The current proposal [1] sets out to segment our different types
> >>>>> of test by placing them in different file locations. After looking
> >>>>> at the recent changes to the LUNI module tests (where the layout
> >>>>> guidelines were applied) I have a real concern that there are
> >>>>> serious problems with this approach. We have started down a track
> >>>>> of just continually growing the number of test source folders as
> >>>>> new categories of test are identified and IMHO that is going to
> >>>>> bring complexity and maintenance issues with these tests.
> >>>>>
> >>>>> Consider the dimensions of tests that we have ...
> >>>>>
> >>>>> API
> >>>>> Harmony-specific
> >>>>> Platform-specific
> >>>>> Run on classpath
> >>>>> Run on bootclasspath
> >>>>> Behaves different between Harmony and RI
> >>>>> Stress
> >>>>> ...and so on...
> >>>>>
> >>>>>
> >>>>> If you weigh up all of the different possible permutations and
> >>>>> then consider that the above list is highly likely to be extended
> >>>>> as things progress it is obvious that we are eventually heading
> >>>>> for large amounts of related test code scattered or possibly
> >>>>> duplicated across numerous "hard wired" source directories. How
> >>>>> maintainable is that going to be ?
> >>>>>
> >>>>> If we want to run different tests in different configurations then
> >>>>> IMHO we need to be thinking a whole lot smarter. We need to be
> >>>>> thinking about keeping tests for specific areas of functionality
> >>>>> together (thus easing maintenance); we need something quick and
> >>>>> simple to re-configure if necessary (pushing whole directories of
> >>>>> files around the place does not seem a particularly lightweight
> >>>>> approach); and something that is not going to potentially mess up
> >>>>> contributed patches when the file they patch is found to have been
> >>>>> recently pushed from source folder A to B.
> >>>>>
> >>>>> To connect into another recent thread, there have been some posts
> >>>>> lately about handling some test methods that fail on Harmony and
> >>>>> have meant that entire test case classes have been excluded from
> >>>>> our test runs. I have also been noticing some API test methods
> >>>>> that pass fine on Harmony but fail when run against the RI. Are
> >>>>> the different behaviours down to errors in the Harmony
> >>>>> implementation ? An error in the RI implementation ? A bug in the
> >>>>> RI Javadoc ? Only after some investigation has been carried out
do
> >>>>> we know for sure. That takes time. What do we do with the test
> >>>>> methods in the meantime ? Do we push them round the file system
> >>>>> into yet another new source folder ? IMHO we need a testing
> >>>>> strategy that enables such "problem" methods to be tracked easily
> >>>>> without disruption to the rest of the other tests.
> >>>>>
> >>>>> A couple of weeks ago I mentioned that the TestNG framework [2]
> >>>>> seemed like a reasonably good way of allowing us to both group
> >>>>> together different kinds of tests and permit the exclusion of
> >>>>> individual tests/groups of tests [3]. I would like to strongly
> >>>>> propose that we consider using TestNG as a means of providing the
> >>>>> different test configurations required by Harmony. Using a
> >>>>> combination of annotations and XML to capture the kinds of
> >>>>> sophisticated test configurations that people need, and that
> >>>>> allows us to specify down to the individual method, has got to be
> >>>>> more scalable and flexible than where we are headed now.
> >>>>>
> >>>>> Thanks for reading this far.
> >>>>>
> >>>>> Best regards,
> >>>>> George
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>>
> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> >>>>>
> >>>>> [2] http://testng.org
> >>>>> [3]
> >>>>>
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e
> >>>>>
>
>


-- 
Thanks,
Stepan Mishura
Intel Middleware Products Division

------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message