harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulex Yang <paulex.y...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Wed, 26 Jul 2006 11:10:58 GMT
FYI, I haven't studied it yet, but seems new TestNG 5 support ant task 
with JVM parameter[1]

[1] http://www.theserverside.com/news/thread.tss?thread_id=41479

Richard Liang wrote:
> Just thinking about using TestNG to execute Harmony test cases.  :-)
>
> Look at our build.xml (e.g., modules/luni/build.xml), you will see 
> something like:
> ..................
> <junit fork="yes"
>                   forkmode="once"
>                   printsummary="withOutAndErr"
>                   errorproperty="test.errors"
>                   failureproperty="test.failures"
>                   showoutput="on"
>                   dir="${hy.luni.bin.test}"
>                   jvm="${test.jre.home}/bin/java">
>
>                <jvmarg value="-showversion" />
> ......
>
> My question is the TestNG Ant task <testng> does not support 
> attributes "fork" and "jvm", how to run our test under Harmony VM? 
> Thanks a lot.
>
> Best regards,
> Richard
>
> Alexei Zakharov wrote:
>> Hmm, do we have problems with launching ant? I thought we have
>> problems with launching TestNG. Just checked - running tests for beans
>> on j9+fresh classlib works fine. I.e.
>> ant -Dbuild.module=beans
>> -Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter test
>>
>> 2006/7/19, Richard Liang <richard.liangyx@gmail.com>:
>>> According to "TestNG Ant Task" [1], it seems that the TestNG Ant task
>>> does not support to fork a new JVM, that is, we must launch ant using
>>> Harmony itself. Any comments? Thanks a lot.
>>>
>>> [1]http://testng.org/doc/ant.html
>>>
>>> Best regards,
>>> Richard
>>>
>>> George Harley wrote:
>>> > Andrew Zhang wrote:
>>> >> On 7/18/06, George Harley <george.c.harley@googlemail.com> wrote:
>>> >>>
>>> >>> Oliver Deakin wrote:
>>> >>> > George Harley wrote:
>>> >>> >> <SNIP!>
>>> >>> >>
>>> >>> >> Here the annotation on MyTestClass applies to all of its
test
>>> >>> methods.
>>> >>> >>
>>> >>> >> So what are the well-known TestNG groups that we could
define 
>>> for
>>> >>> use
>>> >>> >> inside Harmony ? Here are some of my initial thoughts:
>>> >>> >>
>>> >>> >>
>>> >>> >> * type.impl  --  tests that are specific to Harmony
>>> >>> >
>>> >>> > So tests are implicitly API unless specified otherwise?
>>> >>> >
>>> >>> > I'm slightly confused by your definition of impl tests as 
>>> "tests that
>>> >>> are
>>> >>> > specific to Harmony". Does this mean that impl tests are only
>>> >>> > those that test classes in org.apache.harmony packages?
>>> >>> > I thought that impl was our way of saying "tests that need
to 
>>> go on
>>> >>> > the bootclasspath".
>>> >>> >
>>> >>> > I think I just need a little clarification...
>>> >>> >
>>> >>>
>>> >>> Hi Oliver,
>>> >>>
>>> >>> I was using the definition of implementation-specific tests that
we
>>> >>> currently have on the Harmony testing conventions web page. That

>>> is,
>>> >>> implementation-specific tests are those that are dependent on some
>>> >>> aspect of the Harmony implementation and would therefore not 
>>> pass when
>>> >>> run against the RI or other conforming implementations. It's 
>>> orthogonal
>>> >>> to the classpath/bootclasspath issue.
>>> >>>
>>> >>>
>>> >>> >> * state.broken.<platform id>  --  tests bust on a
specific 
>>> platform
>>> >>> >>
>>> >>> >> * state.broken  --  tests broken on every platform but
we 
>>> want to
>>> >>> >> decide whether or not to run from our suite configuration
>>> >>> >>
>>> >>> >> * os.<platform id>  --  tests that are to be run
only on the
>>> >>> >> specified platform (a test could be member of more than
one of
>>> >>> these)
>>> >>> >
>>> >>> > And the defaults for these are an unbroken state and runs on
any
>>> >>> > platform.
>>> >>> > That makes sense...
>>> >>> >
>>> >>> > Will the platform ids be organised in a similar way to the
>>> >>> platform ids
>>> >>> > we've discussed before for organisation of native code [1]?
>>> >>> >
>>> >>>
>>> >>> The actual string used to identify a particular platform can be
>>> >>> whatever
>>> >>> we want it to be, just so long as we are consistent. So, yes, 
>>> the ids
>>> >>> mentioned in the referenced email would seem a good starting 
>>> point. Do
>>> >>> we need to include a 32-bit/64-bit identifier ?
>>> >>>
>>> >>>
>>> >>> > So all tests are, by default, in an all-platforms (or shared)

>>> group.
>>> >>> > If a test fails on all Windows platforms, it is marked with
>>> >>> > state.broken.windows.
>>> >>> > If a test fails on Windows but only on, say, amd hardware,
>>> >>> > it is marked state.broken.windows.amd.
>>> >>> >
>>> >>>
>>> >>> Yes. Agreed.
>>> >>>
>>> >>>
>>> >>> > Then when you come to run tests on your windows amd machine,
>>> >>> > you want to include all tests in the all-platform (shared)
group,
>>> >>> > os.windows and os.windows.amd, and exclude all tests in
>>> >>> > the state.broken, state.broken.windows and 
>>> state.broken.windows.amd
>>> >>> > groups.
>>> >>> >
>>> >>> > Does this tally with what you were thinking?
>>> >>> >
>>> >>>
>>> >>> Yes, that is the idea.
>>> >>>
>>> >>>
>>> >>> >>
>>> >>> >>
>>> >>> >> What does everyone else think ? Does such a scheme sound
>>> >>> reasonable ?
>>> >>> >
>>> >>> > I think so - it seems to cover our current requirements. 
>>> Thanks for
>>> >>> > coming up with this!
>>> >>> >
>>> >>>
>>> >>> Thanks, but I don't see it as final yet really. It would be 
>>> great to
>>> >>> prove the worth of this by doing a trial on one of the existing
>>> >>> modules,
>>> >>> ideally something that contains tests that are platform-specific.
>>> >>
>>> >>
>>> >> Hello George, how about doing a trial on NIO module?
>>> >>
>>> >> So far as I know, there are several platform dependent tests in NIO
>>> >> module.
>>> >> :)
>>> >>
>>> >> The assert statements are commented out in these tests, with "FIXME"
>>> >> mark.
>>> >>
>>> >> Furthurmore, I also find some platform dependent behaviours of
>>> >> FileChannel.
>>> >> If TestNG is applied on NIO, I will supplement new tests for
>>> >> FileChannel and
>>> >> fix the bug of source code.
>>> >>
>>> >> What's your opnion? Any suggestions/comments?
>>> >>
>>> >> Thanks!
>>> >>
>>> >
>>> > Hi Andrew,
>>> >
>>> > That sounds like a very good idea. If there is agreement in the
>>> > project that 5.0 annotations are the way to go (as opposed to the
>>> > pre-5.0 Javadoc comment support offered by TestNG) then to the 
>>> best of
>>> > my knowledge all that is stopping us from doing this trial is the 
>>> lack
>>> > of a 5.0 VM to run the Harmony tests on. Hopefully that will be
>>> > addressed soon. When it is I would be happy to get stuck into this 
>>> trial.
>>> >
>>> > Best regards,
>>> > George
>>> >
>>> >
>>> >> Best regards,
>>> >>> George
>>> >>>
>>> >>>
>>> >>> > Regards,
>>> >>> > Oliver
>>> >>> >
>>> >>> > [1]
>>> >>> >
>>> >>> 
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>>>
>>> >>>
>>> >>> >
>>> >>> >
>>> >>> >>
>>> >>> >> Thanks for reading this far.
>>> >>> >>
>>> >>> >> Best regards,
>>> >>> >> George
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> George Harley wrote:
>>> >>> >>> Hi,
>>> >>> >>>
>>> >>> >>> Just seen Tim's note on test support classes and it
really
>>> >>> caught my
>>> >>> >>> attention as I have been mulling over this issue for
a 
>>> little while
>>> >>> >>> now. I think that it is a good time for us to return
to the
>>> >>> topic of
>>> >>> >>> class library test layouts.
>>> >>> >>>
>>> >>> >>> The current proposal [1] sets out to segment our different

>>> types of
>>> >>> >>> test by placing them in different file locations. After

>>> looking at
>>> >>> >>> the recent changes to the LUNI module tests (where
the layout
>>> >>> >>> guidelines were applied) I have a real concern that
there are
>>> >>> >>> serious problems with this approach. We have started
down a
>>> >>> track of
>>> >>> >>> just continually growing the number of test source
folders 
>>> as new
>>> >>> >>> categories of test are identified and IMHO that is
going to 
>>> bring
>>> >>> >>> complexity and maintenance issues with these tests.
>>> >>> >>>
>>> >>> >>> Consider the dimensions of tests that we have ...
>>> >>> >>>
>>> >>> >>> API
>>> >>> >>> Harmony-specific
>>> >>> >>> Platform-specific
>>> >>> >>> Run on classpath
>>> >>> >>> Run on bootclasspath
>>> >>> >>> Behaves different between Harmony and RI
>>> >>> >>> Stress
>>> >>> >>> ...and so on...
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> If you weigh up all of the different possible permutations

>>> and then
>>> >>> >>> consider that the above list is highly likely to be
extended as
>>> >>> >>> things progress it is obvious that we are eventually
heading 
>>> for
>>> >>> >>> large amounts of related test code scattered or possibly

>>> duplicated
>>> >>> >>> across numerous "hard wired" source directories. How
>>> >>> maintainable is
>>> >>> >>> that going to be ?
>>> >>> >>>
>>> >>> >>> If we want to run different tests in different 
>>> configurations then
>>> >>> >>> IMHO we need to be thinking a whole lot smarter. We
need to be
>>> >>> >>> thinking about keeping tests for specific areas of

>>> functionality
>>> >>> >>> together (thus easing maintenance); we need something
quick and
>>> >>> >>> simple to re-configure if necessary (pushing whole

>>> directories of
>>> >>> >>> files around the place does not seem a particularly
lightweight
>>> >>> >>> approach); and something that is not going to potentially

>>> mess up
>>> >>> >>> contributed patches when the file they patch is found
to 
>>> have been
>>> >>> >>> recently pushed from source folder A to B.
>>> >>> >>>
>>> >>> >>> To connect into another recent thread, there have been
some 
>>> posts
>>> >>> >>> lately about handling some test methods that fail on
Harmony 
>>> and
>>> >>> >>> have meant that entire test case classes have been
excluded 
>>> from
>>> >>> our
>>> >>> >>> test runs. I have also been noticing some API test
methods that
>>> >>> pass
>>> >>> >>> fine on Harmony but fail when run against the RI. Are
the 
>>> different
>>> >>> >>> behaviours down to errors in the Harmony implementation
? An 
>>> error
>>> >>> >>> in the RI implementation ? A bug in the RI Javadoc
? Only after
>>> >>> some
>>> >>> >>> investigation has been carried out do we know for sure.
That 
>>> takes
>>> >>> >>> time. What do we do with the test methods in the meantime
? 
>>> Do we
>>> >>> >>> push them round the file system into yet another new
source
>>> >>> folder ?
>>> >>> >>> IMHO we need a testing strategy that enables such "problem"

>>> methods
>>> >>> >>> to be tracked easily without disruption to the rest
of the 
>>> other
>>> >>> tests.
>>> >>> >>>
>>> >>> >>> A couple of weeks ago I mentioned that the TestNG framework
[2]
>>> >>> >>> seemed like a reasonably good way of allowing us to
both group
>>> >>> >>> together different kinds of tests and permit the exclusion
of
>>> >>> >>> individual tests/groups of tests [3]. I would like
to strongly
>>> >>> >>> propose that we consider using TestNG as a means of

>>> providing the
>>> >>> >>> different test configurations required by Harmony.
Using a
>>> >>> >>> combination of annotations and XML to capture the kinds
of
>>> >>> >>> sophisticated test configurations that people need,
and that 
>>> allows
>>> >>> >>> us to specify down to the individual method, has got
to be more
>>> >>> >>> scalable and flexible than where we are headed now.
>>> >>> >>>
>>> >>> >>> Thanks for reading this far.
>>> >>> >>>
>>> >>> >>> Best regards,
>>> >>> >>> George
>>> >>> >>>
>>> >>> >>>
>>> >>> >>> [1]
>>> >>> >>>
>>> >>> 
>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>>
>>> >>>
>>> >>> >>>
>>> >>> >>> [2] http://testng.org
>>> >>> >>> [3]
>>> >>> >>>
>>> >>> 
>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>>
>>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> 
>>> ---------------------------------------------------------------------
>>> >>> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> >>> >> To unsubscribe, e-mail: 
>>> harmony-dev-unsubscribe@incubator.apache.org
>>> >>> >> For additional commands, e-mail:
>>> >>> harmony-dev-help@incubator.apache.org
>>> >>> >>
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>>
>>> >>> 
>>> ---------------------------------------------------------------------
>>> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> >>> To unsubscribe, e-mail: 
>>> harmony-dev-unsubscribe@incubator.apache.org
>>> >>> For additional commands, e-mail: 
>>> harmony-dev-help@incubator.apache.org
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> > For additional commands, e-mail: 
>>> harmony-dev-help@incubator.apache.org
>>> >
>>> >
>>>
>>> -- 
>>> Richard Liang
>>> China Software Development Lab, IBM
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>>
>>>
>>
>>
>


-- 
Paulex Yang
China Software Development Lab
IBM



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message