harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Wed, 19 Jul 2006 09:56:27 GMT
Just thinking about using TestNG to execute Harmony test cases.  :-)

Look at our build.xml (e.g., modules/luni/build.xml), you will see 
something like:
..................
<junit fork="yes"
                   forkmode="once"
                   printsummary="withOutAndErr"
                   errorproperty="test.errors"
                   failureproperty="test.failures"
                   showoutput="on"
                   dir="${hy.luni.bin.test}"
                   jvm="${test.jre.home}/bin/java">

                <jvmarg value="-showversion" />
......

My question is the TestNG Ant task <testng> does not support attributes 
"fork" and "jvm", how to run our test under Harmony VM? Thanks a lot.

Best regards,
Richard

Alexei Zakharov wrote:
> Hmm, do we have problems with launching ant? I thought we have
> problems with launching TestNG. Just checked - running tests for beans
> on j9+fresh classlib works fine. I.e.
> ant -Dbuild.module=beans
> -Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter test
>
> 2006/7/19, Richard Liang <richard.liangyx@gmail.com>:
>> According to "TestNG Ant Task" [1], it seems that the TestNG Ant task
>> does not support to fork a new JVM, that is, we must launch ant using
>> Harmony itself. Any comments? Thanks a lot.
>>
>> [1]http://testng.org/doc/ant.html
>>
>> Best regards,
>> Richard
>>
>> George Harley wrote:
>> > Andrew Zhang wrote:
>> >> On 7/18/06, George Harley <george.c.harley@googlemail.com> wrote:
>> >>>
>> >>> Oliver Deakin wrote:
>> >>> > George Harley wrote:
>> >>> >> <SNIP!>
>> >>> >>
>> >>> >> Here the annotation on MyTestClass applies to all of its test
>> >>> methods.
>> >>> >>
>> >>> >> So what are the well-known TestNG groups that we could define
for
>> >>> use
>> >>> >> inside Harmony ? Here are some of my initial thoughts:
>> >>> >>
>> >>> >>
>> >>> >> * type.impl  --  tests that are specific to Harmony
>> >>> >
>> >>> > So tests are implicitly API unless specified otherwise?
>> >>> >
>> >>> > I'm slightly confused by your definition of impl tests as 
>> "tests that
>> >>> are
>> >>> > specific to Harmony". Does this mean that impl tests are only
>> >>> > those that test classes in org.apache.harmony packages?
>> >>> > I thought that impl was our way of saying "tests that need to 
>> go on
>> >>> > the bootclasspath".
>> >>> >
>> >>> > I think I just need a little clarification...
>> >>> >
>> >>>
>> >>> Hi Oliver,
>> >>>
>> >>> I was using the definition of implementation-specific tests that we
>> >>> currently have on the Harmony testing conventions web page. That is,
>> >>> implementation-specific tests are those that are dependent on some
>> >>> aspect of the Harmony implementation and would therefore not pass 
>> when
>> >>> run against the RI or other conforming implementations. It's 
>> orthogonal
>> >>> to the classpath/bootclasspath issue.
>> >>>
>> >>>
>> >>> >> * state.broken.<platform id>  --  tests bust on a specific

>> platform
>> >>> >>
>> >>> >> * state.broken  --  tests broken on every platform but we want
to
>> >>> >> decide whether or not to run from our suite configuration
>> >>> >>
>> >>> >> * os.<platform id>  --  tests that are to be run only
on the
>> >>> >> specified platform (a test could be member of more than one
of
>> >>> these)
>> >>> >
>> >>> > And the defaults for these are an unbroken state and runs on any
>> >>> > platform.
>> >>> > That makes sense...
>> >>> >
>> >>> > Will the platform ids be organised in a similar way to the
>> >>> platform ids
>> >>> > we've discussed before for organisation of native code [1]?
>> >>> >
>> >>>
>> >>> The actual string used to identify a particular platform can be
>> >>> whatever
>> >>> we want it to be, just so long as we are consistent. So, yes, the 
>> ids
>> >>> mentioned in the referenced email would seem a good starting 
>> point. Do
>> >>> we need to include a 32-bit/64-bit identifier ?
>> >>>
>> >>>
>> >>> > So all tests are, by default, in an all-platforms (or shared) 
>> group.
>> >>> > If a test fails on all Windows platforms, it is marked with
>> >>> > state.broken.windows.
>> >>> > If a test fails on Windows but only on, say, amd hardware,
>> >>> > it is marked state.broken.windows.amd.
>> >>> >
>> >>>
>> >>> Yes. Agreed.
>> >>>
>> >>>
>> >>> > Then when you come to run tests on your windows amd machine,
>> >>> > you want to include all tests in the all-platform (shared) group,
>> >>> > os.windows and os.windows.amd, and exclude all tests in
>> >>> > the state.broken, state.broken.windows and 
>> state.broken.windows.amd
>> >>> > groups.
>> >>> >
>> >>> > Does this tally with what you were thinking?
>> >>> >
>> >>>
>> >>> Yes, that is the idea.
>> >>>
>> >>>
>> >>> >>
>> >>> >>
>> >>> >> What does everyone else think ? Does such a scheme sound
>> >>> reasonable ?
>> >>> >
>> >>> > I think so - it seems to cover our current requirements. Thanks

>> for
>> >>> > coming up with this!
>> >>> >
>> >>>
>> >>> Thanks, but I don't see it as final yet really. It would be great to
>> >>> prove the worth of this by doing a trial on one of the existing
>> >>> modules,
>> >>> ideally something that contains tests that are platform-specific.
>> >>
>> >>
>> >> Hello George, how about doing a trial on NIO module?
>> >>
>> >> So far as I know, there are several platform dependent tests in NIO
>> >> module.
>> >> :)
>> >>
>> >> The assert statements are commented out in these tests, with "FIXME"
>> >> mark.
>> >>
>> >> Furthurmore, I also find some platform dependent behaviours of
>> >> FileChannel.
>> >> If TestNG is applied on NIO, I will supplement new tests for
>> >> FileChannel and
>> >> fix the bug of source code.
>> >>
>> >> What's your opnion? Any suggestions/comments?
>> >>
>> >> Thanks!
>> >>
>> >
>> > Hi Andrew,
>> >
>> > That sounds like a very good idea. If there is agreement in the
>> > project that 5.0 annotations are the way to go (as opposed to the
>> > pre-5.0 Javadoc comment support offered by TestNG) then to the best of
>> > my knowledge all that is stopping us from doing this trial is the lack
>> > of a 5.0 VM to run the Harmony tests on. Hopefully that will be
>> > addressed soon. When it is I would be happy to get stuck into this 
>> trial.
>> >
>> > Best regards,
>> > George
>> >
>> >
>> >> Best regards,
>> >>> George
>> >>>
>> >>>
>> >>> > Regards,
>> >>> > Oliver
>> >>> >
>> >>> > [1]
>> >>> >
>> >>> 
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e

>>
>> >>>
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> Thanks for reading this far.
>> >>> >>
>> >>> >> Best regards,
>> >>> >> George
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> George Harley wrote:
>> >>> >>> Hi,
>> >>> >>>
>> >>> >>> Just seen Tim's note on test support classes and it really
>> >>> caught my
>> >>> >>> attention as I have been mulling over this issue for a
little 
>> while
>> >>> >>> now. I think that it is a good time for us to return to
the
>> >>> topic of
>> >>> >>> class library test layouts.
>> >>> >>>
>> >>> >>> The current proposal [1] sets out to segment our different

>> types of
>> >>> >>> test by placing them in different file locations. After

>> looking at
>> >>> >>> the recent changes to the LUNI module tests (where the
layout
>> >>> >>> guidelines were applied) I have a real concern that there
are
>> >>> >>> serious problems with this approach. We have started down
a
>> >>> track of
>> >>> >>> just continually growing the number of test source folders
as 
>> new
>> >>> >>> categories of test are identified and IMHO that is going
to 
>> bring
>> >>> >>> complexity and maintenance issues with these tests.
>> >>> >>>
>> >>> >>> Consider the dimensions of tests that we have ...
>> >>> >>>
>> >>> >>> API
>> >>> >>> Harmony-specific
>> >>> >>> Platform-specific
>> >>> >>> Run on classpath
>> >>> >>> Run on bootclasspath
>> >>> >>> Behaves different between Harmony and RI
>> >>> >>> Stress
>> >>> >>> ...and so on...
>> >>> >>>
>> >>> >>>
>> >>> >>> If you weigh up all of the different possible permutations

>> and then
>> >>> >>> consider that the above list is highly likely to be extended
as
>> >>> >>> things progress it is obvious that we are eventually heading
for
>> >>> >>> large amounts of related test code scattered or possibly

>> duplicated
>> >>> >>> across numerous "hard wired" source directories. How
>> >>> maintainable is
>> >>> >>> that going to be ?
>> >>> >>>
>> >>> >>> If we want to run different tests in different configurations

>> then
>> >>> >>> IMHO we need to be thinking a whole lot smarter. We need
to be
>> >>> >>> thinking about keeping tests for specific areas of functionality
>> >>> >>> together (thus easing maintenance); we need something quick
and
>> >>> >>> simple to re-configure if necessary (pushing whole 
>> directories of
>> >>> >>> files around the place does not seem a particularly lightweight
>> >>> >>> approach); and something that is not going to potentially

>> mess up
>> >>> >>> contributed patches when the file they patch is found to
have 
>> been
>> >>> >>> recently pushed from source folder A to B.
>> >>> >>>
>> >>> >>> To connect into another recent thread, there have been
some 
>> posts
>> >>> >>> lately about handling some test methods that fail on Harmony
and
>> >>> >>> have meant that entire test case classes have been excluded
from
>> >>> our
>> >>> >>> test runs. I have also been noticing some API test methods
that
>> >>> pass
>> >>> >>> fine on Harmony but fail when run against the RI. Are the

>> different
>> >>> >>> behaviours down to errors in the Harmony implementation
? An 
>> error
>> >>> >>> in the RI implementation ? A bug in the RI Javadoc ? Only
after
>> >>> some
>> >>> >>> investigation has been carried out do we know for sure.
That 
>> takes
>> >>> >>> time. What do we do with the test methods in the meantime
? 
>> Do we
>> >>> >>> push them round the file system into yet another new source
>> >>> folder ?
>> >>> >>> IMHO we need a testing strategy that enables such "problem"

>> methods
>> >>> >>> to be tracked easily without disruption to the rest of
the other
>> >>> tests.
>> >>> >>>
>> >>> >>> A couple of weeks ago I mentioned that the TestNG framework
[2]
>> >>> >>> seemed like a reasonably good way of allowing us to both
group
>> >>> >>> together different kinds of tests and permit the exclusion
of
>> >>> >>> individual tests/groups of tests [3]. I would like to strongly
>> >>> >>> propose that we consider using TestNG as a means of providing

>> the
>> >>> >>> different test configurations required by Harmony. Using
a
>> >>> >>> combination of annotations and XML to capture the kinds
of
>> >>> >>> sophisticated test configurations that people need, and
that 
>> allows
>> >>> >>> us to specify down to the individual method, has got to
be more
>> >>> >>> scalable and flexible than where we are headed now.
>> >>> >>>
>> >>> >>> Thanks for reading this far.
>> >>> >>>
>> >>> >>> Best regards,
>> >>> >>> George
>> >>> >>>
>> >>> >>>
>> >>> >>> [1]
>> >>> >>>
>> >>> 
>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html 
>>
>> >>>
>> >>> >>>
>> >>> >>> [2] http://testng.org
>> >>> >>> [3]
>> >>> >>>
>> >>> 
>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e

>>
>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> 
>> ---------------------------------------------------------------------
>> >>> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >>> >> To unsubscribe, e-mail: 
>> harmony-dev-unsubscribe@incubator.apache.org
>> >>> >> For additional commands, e-mail:
>> >>> harmony-dev-help@incubator.apache.org
>> >>> >>
>> >>> >>
>> >>> >
>> >>>
>> >>>
>> >>> 
>> ---------------------------------------------------------------------
>> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> >>> For additional commands, e-mail: 
>> harmony-dev-help@incubator.apache.org
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>> -- 
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message