harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Thu, 20 Jul 2006 02:17:18 GMT


George Harley wrote:
> Hi Alexei,
>
> It's encouraging to hear that (Ant + TestNG + sample tests) all worked 
> fine together on Harmony. In answer to your question I suppose that 
> the ability to fork the tests in a separate VM means that we do not 
> run the risk of possible bugs in Harmony affecting the test harness 
> and therefore the outcome of the tests.
>
Do you think it's reasonable to launch ant using Harmony, and then 
TestNG will fork another Harmony VM to run our tests? Thanks a lot.

Best regards,
Richard.

> Best regards,
> George
>
>
> Alexei Zakharov wrote:
>> Probably my previous message was not clear enough.
>> Why can't we just invoke everything including ant on top of Harmony
>> for now? At least I was able to build and run test-14 examples from
>> TestNG 4.7 distribution solely on top of j9 + our classlib today.
>>
>> C:\Java\testng-4.7\test-14>set 
>> JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk
>> \deploy\jdk\jre
>>
>> C:\Java\testng-4.7\test-14>ant 
>> -Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler
>> Adapter run
>> Buildfile: build.xml
>>
>> prepare:
>>
>> compile:
>>     [echo]                                  -- Compiling JDK 1.4 
>> tests --
>>
>> run:
>>     [echo]                                  -- Running JDK 1.4 
>> tests   --
>>     [echo]                                  -- testng-4.7-jdk14.jar  --
>>
>> [testng-14] ===============================================
>> [testng-14] TestNG JDK 1.4
>> [testng-14] Total tests run: 179, Failures: 10, Skips: 0
>> [testng-14] ===============================================
>> ...
>>
>> Exactly the same results as with Sun JDK 1.4.
>> Note: you may need to hatch the build.xml a little bit to achieve this.
>>
>> Thanks,
>>
>> 2006/7/19, George Harley <george.c.harley@googlemail.com>:
>>> Hi Richard,
>>>
>>> Actually the Ant task always runs the tests in a forked VM. At present,
>>> however, the task does not support specifying the forked VM (i.e. there
>>> is no equivalent to the JUnit Ant task's "jvm" attribute). This matter
>>> has already been raised with the TestNG folks who seem happy to
>>> introduce this.
>>>
>>> In the meantime we could run the tests using the Ant java task.
>>>
>>>
>>> Best regards,
>>> George
>>>
>>>
>>>
>>> Richard Liang wrote:
>>> > According to "TestNG Ant Task" [1], it seems that the TestNG Ant task
>>> > does not support to fork a new JVM, that is, we must launch ant using
>>> > Harmony itself. Any comments? Thanks a lot.
>>> >
>>> > [1]http://testng.org/doc/ant.html
>>> >
>>> > Best regards,
>>> > Richard
>>> >
>>> > George Harley wrote:
>>> >> Andrew Zhang wrote:
>>> >>> On 7/18/06, George Harley <george.c.harley@googlemail.com>
wrote:
>>> >>>>
>>> >>>> Oliver Deakin wrote:
>>> >>>> > George Harley wrote:
>>> >>>> >> <SNIP!>
>>> >>>> >>
>>> >>>> >> Here the annotation on MyTestClass applies to all of
its test
>>> >>>> methods.
>>> >>>> >>
>>> >>>> >> So what are the well-known TestNG groups that we could
define
>>> >>>> for use
>>> >>>> >> inside Harmony ? Here are some of my initial thoughts:
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> * type.impl  --  tests that are specific to Harmony
>>> >>>> >
>>> >>>> > So tests are implicitly API unless specified otherwise?
>>> >>>> >
>>> >>>> > I'm slightly confused by your definition of impl tests
as "tests
>>> >>>> that
>>> >>>> are
>>> >>>> > specific to Harmony". Does this mean that impl tests are
only
>>> >>>> > those that test classes in org.apache.harmony packages?
>>> >>>> > I thought that impl was our way of saying "tests that need
to 
>>> go on
>>> >>>> > the bootclasspath".
>>> >>>> >
>>> >>>> > I think I just need a little clarification...
>>> >>>> >
>>> >>>>
>>> >>>> Hi Oliver,
>>> >>>>
>>> >>>> I was using the definition of implementation-specific tests

>>> that we
>>> >>>> currently have on the Harmony testing conventions web page.

>>> That is,
>>> >>>> implementation-specific tests are those that are dependent on
some
>>> >>>> aspect of the Harmony implementation and would therefore not

>>> pass when
>>> >>>> run against the RI or other conforming implementations. It's
>>> >>>> orthogonal
>>> >>>> to the classpath/bootclasspath issue.
>>> >>>>
>>> >>>>
>>> >>>> >> * state.broken.<platform id>  --  tests bust
on a specific 
>>> platform
>>> >>>> >>
>>> >>>> >> * state.broken  --  tests broken on every platform
but we 
>>> want to
>>> >>>> >> decide whether or not to run from our suite configuration
>>> >>>> >>
>>> >>>> >> * os.<platform id>  --  tests that are to be
run only on the
>>> >>>> >> specified platform (a test could be member of more
than one of
>>> >>>> these)
>>> >>>> >
>>> >>>> > And the defaults for these are an unbroken state and runs
on any
>>> >>>> > platform.
>>> >>>> > That makes sense...
>>> >>>> >
>>> >>>> > Will the platform ids be organised in a similar way to
the
>>> >>>> platform ids
>>> >>>> > we've discussed before for organisation of native code
[1]?
>>> >>>> >
>>> >>>>
>>> >>>> The actual string used to identify a particular platform can
be
>>> >>>> whatever
>>> >>>> we want it to be, just so long as we are consistent. So, yes,

>>> the ids
>>> >>>> mentioned in the referenced email would seem a good starting

>>> point. Do
>>> >>>> we need to include a 32-bit/64-bit identifier ?
>>> >>>>
>>> >>>>
>>> >>>> > So all tests are, by default, in an all-platforms (or shared)

>>> group.
>>> >>>> > If a test fails on all Windows platforms, it is marked
with
>>> >>>> > state.broken.windows.
>>> >>>> > If a test fails on Windows but only on, say, amd hardware,
>>> >>>> > it is marked state.broken.windows.amd.
>>> >>>> >
>>> >>>>
>>> >>>> Yes. Agreed.
>>> >>>>
>>> >>>>
>>> >>>> > Then when you come to run tests on your windows amd machine,
>>> >>>> > you want to include all tests in the all-platform (shared)

>>> group,
>>> >>>> > os.windows and os.windows.amd, and exclude all tests in
>>> >>>> > the state.broken, state.broken.windows and 
>>> state.broken.windows.amd
>>> >>>> > groups.
>>> >>>> >
>>> >>>> > Does this tally with what you were thinking?
>>> >>>> >
>>> >>>>
>>> >>>> Yes, that is the idea.
>>> >>>>
>>> >>>>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> What does everyone else think ? Does such a scheme
sound
>>> >>>> reasonable ?
>>> >>>> >
>>> >>>> > I think so - it seems to cover our current requirements.

>>> Thanks for
>>> >>>> > coming up with this!
>>> >>>> >
>>> >>>>
>>> >>>> Thanks, but I don't see it as final yet really. It would be

>>> great to
>>> >>>> prove the worth of this by doing a trial on one of the existing
>>> >>>> modules,
>>> >>>> ideally something that contains tests that are platform-specific.
>>> >>>
>>> >>>
>>> >>> Hello George, how about doing a trial on NIO module?
>>> >>>
>>> >>> So far as I know, there are several platform dependent tests in
NIO
>>> >>> module.
>>> >>> :)
>>> >>>
>>> >>> The assert statements are commented out in these tests, with 
>>> "FIXME"
>>> >>> mark.
>>> >>>
>>> >>> Furthurmore, I also find some platform dependent behaviours of
>>> >>> FileChannel.
>>> >>> If TestNG is applied on NIO, I will supplement new tests for
>>> >>> FileChannel and
>>> >>> fix the bug of source code.
>>> >>>
>>> >>> What's your opnion? Any suggestions/comments?
>>> >>>
>>> >>> Thanks!
>>> >>>
>>> >>
>>> >> Hi Andrew,
>>> >>
>>> >> That sounds like a very good idea. If there is agreement in the
>>> >> project that 5.0 annotations are the way to go (as opposed to the
>>> >> pre-5.0 Javadoc comment support offered by TestNG) then to the best
>>> >> of my knowledge all that is stopping us from doing this trial is the
>>> >> lack of a 5.0 VM to run the Harmony tests on. Hopefully that will be
>>> >> addressed soon. When it is I would be happy to get stuck into this
>>> >> trial.
>>> >>
>>> >> Best regards,
>>> >> George
>>> >>
>>> >>
>>> >>> Best regards,
>>> >>>> George
>>> >>>>
>>> >>>>
>>> >>>> > Regards,
>>> >>>> > Oliver
>>> >>>> >
>>> >>>> > [1]
>>> >>>> >
>>> >>>> 
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>>>
>>> >>>>
>>> >>>> >
>>> >>>> >
>>> >>>> >>
>>> >>>> >> Thanks for reading this far.
>>> >>>> >>
>>> >>>> >> Best regards,
>>> >>>> >> George
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> George Harley wrote:
>>> >>>> >>> Hi,
>>> >>>> >>>
>>> >>>> >>> Just seen Tim's note on test support classes and
it really
>>> >>>> caught my
>>> >>>> >>> attention as I have been mulling over this issue
for a little
>>> >>>> while
>>> >>>> >>> now. I think that it is a good time for us to return
to the
>>> >>>> topic of
>>> >>>> >>> class library test layouts.
>>> >>>> >>>
>>> >>>> >>> The current proposal [1] sets out to segment our
different
>>> >>>> types of
>>> >>>> >>> test by placing them in different file locations.
After 
>>> looking at
>>> >>>> >>> the recent changes to the LUNI module tests (where
the layout
>>> >>>> >>> guidelines were applied) I have a real concern
that there are
>>> >>>> >>> serious problems with this approach. We have started
down a
>>> >>>> track of
>>> >>>> >>> just continually growing the number of test source
folders 
>>> as new
>>> >>>> >>> categories of test are identified and IMHO that
is going to 
>>> bring
>>> >>>> >>> complexity and maintenance issues with these tests.
>>> >>>> >>>
>>> >>>> >>> Consider the dimensions of tests that we have ...
>>> >>>> >>>
>>> >>>> >>> API
>>> >>>> >>> Harmony-specific
>>> >>>> >>> Platform-specific
>>> >>>> >>> Run on classpath
>>> >>>> >>> Run on bootclasspath
>>> >>>> >>> Behaves different between Harmony and RI
>>> >>>> >>> Stress
>>> >>>> >>> ...and so on...
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>> If you weigh up all of the different possible permutations
and
>>> >>>> then
>>> >>>> >>> consider that the above list is highly likely to
be 
>>> extended as
>>> >>>> >>> things progress it is obvious that we are eventually

>>> heading for
>>> >>>> >>> large amounts of related test code scattered or
possibly
>>> >>>> duplicated
>>> >>>> >>> across numerous "hard wired" source directories.
How
>>> >>>> maintainable is
>>> >>>> >>> that going to be ?
>>> >>>> >>>
>>> >>>> >>> If we want to run different tests in different

>>> configurations then
>>> >>>> >>> IMHO we need to be thinking a whole lot smarter.
We need to be
>>> >>>> >>> thinking about keeping tests for specific areas
of 
>>> functionality
>>> >>>> >>> together (thus easing maintenance); we need something
quick 
>>> and
>>> >>>> >>> simple to re-configure if necessary (pushing whole

>>> directories of
>>> >>>> >>> files around the place does not seem a particularly

>>> lightweight
>>> >>>> >>> approach); and something that is not going to potentially

>>> mess up
>>> >>>> >>> contributed patches when the file they patch is
found to 
>>> have been
>>> >>>> >>> recently pushed from source folder A to B.
>>> >>>> >>>
>>> >>>> >>> To connect into another recent thread, there have
been some 
>>> posts
>>> >>>> >>> lately about handling some test methods that fail
on 
>>> Harmony and
>>> >>>> >>> have meant that entire test case classes have been
excluded
>>> >>>> from our
>>> >>>> >>> test runs. I have also been noticing some API test
methods 
>>> that
>>> >>>> pass
>>> >>>> >>> fine on Harmony but fail when run against the RI.
Are the
>>> >>>> different
>>> >>>> >>> behaviours down to errors in the Harmony implementation
? 
>>> An error
>>> >>>> >>> in the RI implementation ? A bug in the RI Javadoc
? Only 
>>> after
>>> >>>> some
>>> >>>> >>> investigation has been carried out do we know for
sure. 
>>> That takes
>>> >>>> >>> time. What do we do with the test methods in the
meantime ? 
>>> Do we
>>> >>>> >>> push them round the file system into yet another
new source
>>> >>>> folder ?
>>> >>>> >>> IMHO we need a testing strategy that enables such
"problem"
>>> >>>> methods
>>> >>>> >>> to be tracked easily without disruption to the
rest of the 
>>> other
>>> >>>> tests.
>>> >>>> >>>
>>> >>>> >>> A couple of weeks ago I mentioned that the TestNG
framework 
>>> [2]
>>> >>>> >>> seemed like a reasonably good way of allowing us
to both group
>>> >>>> >>> together different kinds of tests and permit the
exclusion of
>>> >>>> >>> individual tests/groups of tests [3]. I would like
to strongly
>>> >>>> >>> propose that we consider using TestNG as a means
of 
>>> providing the
>>> >>>> >>> different test configurations required by Harmony.
Using a
>>> >>>> >>> combination of annotations and XML to capture the
kinds of
>>> >>>> >>> sophisticated test configurations that people need,
and that
>>> >>>> allows
>>> >>>> >>> us to specify down to the individual method, has
got to be 
>>> more
>>> >>>> >>> scalable and flexible than where we are headed
now.
>>> >>>> >>>
>>> >>>> >>> Thanks for reading this far.
>>> >>>> >>>
>>> >>>> >>> Best regards,
>>> >>>> >>> George
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>> [1]
>>> >>>> >>>
>>> >>>> 
>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>>
>>> >>>>
>>> >>>> >>>
>>> >>>> >>> [2] http://testng.org
>>> >>>> >>> [3]
>>> >>>> >>>
>>> >>>> 
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>>
>>> >>>>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> 
>>> ---------------------------------------------------------------------
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message