harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [classlib] Testing conventions - a proposal
Date Wed, 19 Jul 2006 11:01:11 GMT
Probably my previous message was not clear enough.
Why can't we just invoke everything including ant on top of Harmony
for now? At least I was able to build and run test-14 examples from
TestNG 4.7 distribution solely on top of j9 + our classlib today.

C:\Java\testng-4.7\test-14>set JAVA_HOME=c:\Java\harmony\enhanced\classlib\trunk
\deploy\jdk\jre

C:\Java\testng-4.7\test-14>ant -Dbuild.compiler=org.eclipse.jdt.core.JDTCompiler
Adapter run
Buildfile: build.xml

prepare:

compile:
     [echo]                                  -- Compiling JDK 1.4 tests --

run:
     [echo]                                  -- Running JDK 1.4 tests   --
     [echo]                                  -- testng-4.7-jdk14.jar  --

[testng-14] ===============================================
[testng-14] TestNG JDK 1.4
[testng-14] Total tests run: 179, Failures: 10, Skips: 0
[testng-14] ===============================================
...

Exactly the same results as with Sun JDK 1.4.
Note: you may need to hatch the build.xml a little bit to achieve this.

Thanks,

2006/7/19, George Harley <george.c.harley@googlemail.com>:
> Hi Richard,
>
> Actually the Ant task always runs the tests in a forked VM. At present,
> however, the task does not support specifying the forked VM (i.e. there
> is no equivalent to the JUnit Ant task's "jvm" attribute). This matter
> has already been raised with the TestNG folks who seem happy to
> introduce this.
>
> In the meantime we could run the tests using the Ant java task.
>
>
> Best regards,
> George
>
>
>
> Richard Liang wrote:
> > According to "TestNG Ant Task" [1], it seems that the TestNG Ant task
> > does not support to fork a new JVM, that is, we must launch ant using
> > Harmony itself. Any comments? Thanks a lot.
> >
> > [1]http://testng.org/doc/ant.html
> >
> > Best regards,
> > Richard
> >
> > George Harley wrote:
> >> Andrew Zhang wrote:
> >>> On 7/18/06, George Harley <george.c.harley@googlemail.com> wrote:
> >>>>
> >>>> Oliver Deakin wrote:
> >>>> > George Harley wrote:
> >>>> >> <SNIP!>
> >>>> >>
> >>>> >> Here the annotation on MyTestClass applies to all of its test
> >>>> methods.
> >>>> >>
> >>>> >> So what are the well-known TestNG groups that we could define
> >>>> for use
> >>>> >> inside Harmony ? Here are some of my initial thoughts:
> >>>> >>
> >>>> >>
> >>>> >> * type.impl  --  tests that are specific to Harmony
> >>>> >
> >>>> > So tests are implicitly API unless specified otherwise?
> >>>> >
> >>>> > I'm slightly confused by your definition of impl tests as "tests
> >>>> that
> >>>> are
> >>>> > specific to Harmony". Does this mean that impl tests are only
> >>>> > those that test classes in org.apache.harmony packages?
> >>>> > I thought that impl was our way of saying "tests that need to go
on
> >>>> > the bootclasspath".
> >>>> >
> >>>> > I think I just need a little clarification...
> >>>> >
> >>>>
> >>>> Hi Oliver,
> >>>>
> >>>> I was using the definition of implementation-specific tests that we
> >>>> currently have on the Harmony testing conventions web page. That is,
> >>>> implementation-specific tests are those that are dependent on some
> >>>> aspect of the Harmony implementation and would therefore not pass when
> >>>> run against the RI or other conforming implementations. It's
> >>>> orthogonal
> >>>> to the classpath/bootclasspath issue.
> >>>>
> >>>>
> >>>> >> * state.broken.<platform id>  --  tests bust on a specific
platform
> >>>> >>
> >>>> >> * state.broken  --  tests broken on every platform but we want
to
> >>>> >> decide whether or not to run from our suite configuration
> >>>> >>
> >>>> >> * os.<platform id>  --  tests that are to be run only
on the
> >>>> >> specified platform (a test could be member of more than one
of
> >>>> these)
> >>>> >
> >>>> > And the defaults for these are an unbroken state and runs on any
> >>>> > platform.
> >>>> > That makes sense...
> >>>> >
> >>>> > Will the platform ids be organised in a similar way to the
> >>>> platform ids
> >>>> > we've discussed before for organisation of native code [1]?
> >>>> >
> >>>>
> >>>> The actual string used to identify a particular platform can be
> >>>> whatever
> >>>> we want it to be, just so long as we are consistent. So, yes, the ids
> >>>> mentioned in the referenced email would seem a good starting point.
Do
> >>>> we need to include a 32-bit/64-bit identifier ?
> >>>>
> >>>>
> >>>> > So all tests are, by default, in an all-platforms (or shared) group.
> >>>> > If a test fails on all Windows platforms, it is marked with
> >>>> > state.broken.windows.
> >>>> > If a test fails on Windows but only on, say, amd hardware,
> >>>> > it is marked state.broken.windows.amd.
> >>>> >
> >>>>
> >>>> Yes. Agreed.
> >>>>
> >>>>
> >>>> > Then when you come to run tests on your windows amd machine,
> >>>> > you want to include all tests in the all-platform (shared) group,
> >>>> > os.windows and os.windows.amd, and exclude all tests in
> >>>> > the state.broken, state.broken.windows and state.broken.windows.amd
> >>>> > groups.
> >>>> >
> >>>> > Does this tally with what you were thinking?
> >>>> >
> >>>>
> >>>> Yes, that is the idea.
> >>>>
> >>>>
> >>>> >>
> >>>> >>
> >>>> >> What does everyone else think ? Does such a scheme sound
> >>>> reasonable ?
> >>>> >
> >>>> > I think so - it seems to cover our current requirements. Thanks
for
> >>>> > coming up with this!
> >>>> >
> >>>>
> >>>> Thanks, but I don't see it as final yet really. It would be great to
> >>>> prove the worth of this by doing a trial on one of the existing
> >>>> modules,
> >>>> ideally something that contains tests that are platform-specific.
> >>>
> >>>
> >>> Hello George, how about doing a trial on NIO module?
> >>>
> >>> So far as I know, there are several platform dependent tests in NIO
> >>> module.
> >>> :)
> >>>
> >>> The assert statements are commented out in these tests, with "FIXME"
> >>> mark.
> >>>
> >>> Furthurmore, I also find some platform dependent behaviours of
> >>> FileChannel.
> >>> If TestNG is applied on NIO, I will supplement new tests for
> >>> FileChannel and
> >>> fix the bug of source code.
> >>>
> >>> What's your opnion? Any suggestions/comments?
> >>>
> >>> Thanks!
> >>>
> >>
> >> Hi Andrew,
> >>
> >> That sounds like a very good idea. If there is agreement in the
> >> project that 5.0 annotations are the way to go (as opposed to the
> >> pre-5.0 Javadoc comment support offered by TestNG) then to the best
> >> of my knowledge all that is stopping us from doing this trial is the
> >> lack of a 5.0 VM to run the Harmony tests on. Hopefully that will be
> >> addressed soon. When it is I would be happy to get stuck into this
> >> trial.
> >>
> >> Best regards,
> >> George
> >>
> >>
> >>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>> > Regards,
> >>>> > Oliver
> >>>> >
> >>>> > [1]
> >>>> >
> >>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c44687AAA.5080302@googlemail.com%3e
> >>>>
> >>>> >
> >>>> >
> >>>> >>
> >>>> >> Thanks for reading this far.
> >>>> >>
> >>>> >> Best regards,
> >>>> >> George
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> George Harley wrote:
> >>>> >>> Hi,
> >>>> >>>
> >>>> >>> Just seen Tim's note on test support classes and it really
> >>>> caught my
> >>>> >>> attention as I have been mulling over this issue for a
little
> >>>> while
> >>>> >>> now. I think that it is a good time for us to return to
the
> >>>> topic of
> >>>> >>> class library test layouts.
> >>>> >>>
> >>>> >>> The current proposal [1] sets out to segment our different
> >>>> types of
> >>>> >>> test by placing them in different file locations. After
looking at
> >>>> >>> the recent changes to the LUNI module tests (where the
layout
> >>>> >>> guidelines were applied) I have a real concern that there
are
> >>>> >>> serious problems with this approach. We have started down
a
> >>>> track of
> >>>> >>> just continually growing the number of test source folders
as new
> >>>> >>> categories of test are identified and IMHO that is going
to bring
> >>>> >>> complexity and maintenance issues with these tests.
> >>>> >>>
> >>>> >>> Consider the dimensions of tests that we have ...
> >>>> >>>
> >>>> >>> API
> >>>> >>> Harmony-specific
> >>>> >>> Platform-specific
> >>>> >>> Run on classpath
> >>>> >>> Run on bootclasspath
> >>>> >>> Behaves different between Harmony and RI
> >>>> >>> Stress
> >>>> >>> ...and so on...
> >>>> >>>
> >>>> >>>
> >>>> >>> If you weigh up all of the different possible permutations
and
> >>>> then
> >>>> >>> consider that the above list is highly likely to be extended
as
> >>>> >>> things progress it is obvious that we are eventually heading
for
> >>>> >>> large amounts of related test code scattered or possibly
> >>>> duplicated
> >>>> >>> across numerous "hard wired" source directories. How
> >>>> maintainable is
> >>>> >>> that going to be ?
> >>>> >>>
> >>>> >>> If we want to run different tests in different configurations
then
> >>>> >>> IMHO we need to be thinking a whole lot smarter. We need
to be
> >>>> >>> thinking about keeping tests for specific areas of functionality
> >>>> >>> together (thus easing maintenance); we need something quick
and
> >>>> >>> simple to re-configure if necessary (pushing whole directories
of
> >>>> >>> files around the place does not seem a particularly lightweight
> >>>> >>> approach); and something that is not going to potentially
mess up
> >>>> >>> contributed patches when the file they patch is found to
have been
> >>>> >>> recently pushed from source folder A to B.
> >>>> >>>
> >>>> >>> To connect into another recent thread, there have been
some posts
> >>>> >>> lately about handling some test methods that fail on Harmony
and
> >>>> >>> have meant that entire test case classes have been excluded
> >>>> from our
> >>>> >>> test runs. I have also been noticing some API test methods
that
> >>>> pass
> >>>> >>> fine on Harmony but fail when run against the RI. Are the
> >>>> different
> >>>> >>> behaviours down to errors in the Harmony implementation
? An error
> >>>> >>> in the RI implementation ? A bug in the RI Javadoc ? Only
after
> >>>> some
> >>>> >>> investigation has been carried out do we know for sure.
That takes
> >>>> >>> time. What do we do with the test methods in the meantime
? Do we
> >>>> >>> push them round the file system into yet another new source
> >>>> folder ?
> >>>> >>> IMHO we need a testing strategy that enables such "problem"
> >>>> methods
> >>>> >>> to be tracked easily without disruption to the rest of
the other
> >>>> tests.
> >>>> >>>
> >>>> >>> A couple of weeks ago I mentioned that the TestNG framework
[2]
> >>>> >>> seemed like a reasonably good way of allowing us to both
group
> >>>> >>> together different kinds of tests and permit the exclusion
of
> >>>> >>> individual tests/groups of tests [3]. I would like to strongly
> >>>> >>> propose that we consider using TestNG as a means of providing
the
> >>>> >>> different test configurations required by Harmony. Using
a
> >>>> >>> combination of annotations and XML to capture the kinds
of
> >>>> >>> sophisticated test configurations that people need, and
that
> >>>> allows
> >>>> >>> us to specify down to the individual method, has got to
be more
> >>>> >>> scalable and flexible than where we are headed now.
> >>>> >>>
> >>>> >>> Thanks for reading this far.
> >>>> >>>
> >>>> >>> Best regards,
> >>>> >>> George
> >>>> >>>
> >>>> >>>
> >>>> >>> [1]
> >>>> >>>
> >>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> >>>>
> >>>> >>>
> >>>> >>> [2] http://testng.org
> >>>> >>> [3]
> >>>> >>>
> >>>> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/%3c44A163B3.6080005@googlemail.com%3e
> >>>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> ---------------------------------------------------------------------



-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message