harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [testing] metadata approach
Date Thu, 10 Aug 2006 17:15:47 GMT
Hi Oliver,

> So perhaps the build system should be changed temporarily so that
> we dont self host the test harness? i.e. until we get java.util.concurrent,
> run Ant and the subsequent TestNG process with RI or other non-Harmony
> VM, and launch the tests with Harmony VM using the jvm option.

The bad news is that TestNG requires j.u.c stuff even for single test
execution (i.e. in any case if jvm="<path to harmony>"). :( So if we
want to run annotated tests with TestNG (even from the command line)
we need j.u.c.

> That's odd - Thread.class in luni-kernel (VME v4) definitely contains a
> getId() method.

May be I did something wrong - I will check tomorrow.

> I do like them - in fact I was going to link his mail after that point
> but couldn't find it.

Here is the link:
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44B75E41.7020901@googlemail.com%3e

As far as I remember there was additions to the George's list like
using @Test (groups={"os.any"} ) rather than simple @Test for API
tests that run on any platform.

> I really mean that we should make sure that everyone is happy with
> a certain set of group names before going ahead and applying them.
> Once they are decided upon, they should be added to the testing
> conventions webpage.

Yes, agree.


With Best Regards,

2006/8/10, Oliver Deakin <oliver.deakin@googlemail.com>:
> Alexei Zakharov wrote:
> >> > We now have this, so let the TestNG debate continue :)
> >> Unfortunately, we still need java.util.concurrent :-(
> >
> > Yeah, TestNG 5.0 still throws java.lang.NoClassDefFoundError :
> > java.util.concurrent.LinkedBlockingQueue on Harmony+j9v4.
>
> So perhaps the build system should be changed temporarily so that
> we dont self host the test harness? i.e. until we get java.util.concurrent,
> run Ant and the subsequent TestNG process with RI or other non-Harmony
> VM, and launch the tests with Harmony VM using the jvm option.
> At least it will allow us to move forward with TestNG (if that's what we
> decide) without having to wait for java.util.concurrent. Then when
> we have j.u.c, start self-hosting again.
> Comments?
>
> >
> > I've also got an error while trying to compile TestNG 5.0 tests with
> > Harmony+j9v4+ecj: The method getId() is undefined for the type Thread
>
> That's odd - Thread.class in luni-kernel (VME v4) definitely contains a
> getId() method. I don't know anything about the TestNG tests - how are
> they run? is luni-kernel.jar definitely at the front of the bootclasspath?
>
> >
> >> > - If we go ahead with TestNG, we need to select a set of group names
> >> > to use to indicate exclusion, platform specificness etc.
> >
> > Don't you like the names suggested by George?
>
> I do like them - in fact I was going to link his mail after that point
> but couldn't
> find it. I really mean that we should make sure that everyone is happy with
> a certain set of group names before going ahead and applying them.
> Once they are decided upon, they should be added to the testing
> conventions webpage.
>
> >
> >> > - Decide whether some physical separation of tests on disk is
> >> necessary,  for instance to separate classpath and bootclasspath tests
> >
> > IMHO it is ok to separate classpath and bootclasspath tests because it
> > will be easer to implement such solution technically.
>
> I agree, although I don't feel strongly about it.
>
> Regards,
> Oliver
>
> >
> > Regards,
> >
> > 2006/8/10, Richard Liang <richard.liangyx@gmail.com>:
> >>
> >>
> >> Oliver Deakin wrote:
> >> > Richard Liang wrote:
> >> >>
> >> >>
> >> >> Alexei Zakharov wrote:
> >> >>> Hi Richard,
> >> >>>
> >> >>>> Not sure if we really want to involve another migration: TestNG
> >> >>>> javadoc
> >> >>>> -> TestNG annotation. Any comments?
> >> >>>
> >> >>> Well, IMHO this depends on time constraints - when do we plan to
> >> have
> >> >>> the support for anotations? If the answer is about a couple of
> >> weeks -
> >> >>> no problem, we can wait. But if this is several months...
> >> >>> About the "migration" - I don't think this will be a real painfull
> >> >>> migration, all infrastructure will remain the same. We will only
> >> need
> >> >>> to convert javadocs to annotations (one-one correspondence) and
this
> >> >>> task can be easily automated.
> >> >> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell us when
> >> >> we will have a VM with annotation support?
> >> >
> >> > We now have this, so let the TestNG debate continue :)
> >> >
> >>
> >> Unfortunately, we still need java.util.concurrent :-(
> >>
> >>
> >> > I guess we need to decide a few things before we go ahead with this:
> >> > - Whether TestNG is generally accepted by the Harmony community
> >> > as our test harness of choice for unit testing. I think this will
> >> > probably
> >> > require a vote of some kind before we could make the move.
> >> > - If we go ahead with TestNG, we need to select a set of group names
> >> > to use
> >> > to indicate exclusion, platform specificness etc.
> >> > - Decide whether some physical separation of tests on disk is
> >> necessary,
> >> > for instance to separate classpath and bootclasspath tests.
> >> >
> >> > Comments/additions?
> >>
> >> Agree.  And we could provide proposals for these questions case by case,
> >> and let community make decision.
> >>
> >> Best regards,
> >> Richard
> >> >
> >> > Regards,
> >> > Oliver
> >> >
> >> >
> >> >>
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
> >> >>>>
> >> >>>>
> >> >>>> Alexei Zakharov wrote:
> >> >>>> > Hi,
> >> >>>> >
> >> >>>> > I have created this new thread as a single place for discussions
> >> >>>> > started in "Re: [testing] Peace" and "[classlib] Testing
> >> >>>> conventions –
> >> >>>> > a proposal" threads.
> >> >>>> >
> >> >>>> > What did we have in the previous threads?
> >> >>>> > * Test classification proposed by Vladimir
> >> >>>> > * Test classification and group names proposed by George
> >> >>>> > * Solution for Ant and TestNG scripting issues (still
being
> >> >>>> discussed)
> >> >>>> >
> >> >>>> > Since a lot of people are asking about TestNG and wanting
> >> TestNG I
> >> >>>> > decide to put some effort and take a closer look at this
piece of
> >> >>>> > software. Thus during the last few days I was playing
with TestNG
> >> >>>> - I
> >> >>>> > tried to run different kind of tests with it, to perform
various
> >> >>>> > workloads, generate reports in different ways and etc.
The
> >> >>>> purpose of
> >> >>>> > all this activity was to try TestNG in a real work, understand
is
> >> >>>> > TestNG really worth our credits and how expensive can
be
> >> moving to
> >> >>>> > TestNG from our currently implemented testing infrastructure.
> >> Now I
> >> >>>> > have some thoughts and facts I'd like to share with the
> >> community.
> >> >>>> > I've put it in the form of list for convenience.
> >> >>>> >
> >> >>>> > * TestNG works ok in normal conditions, no visible bugs
> >> >>>> > * It is possible to define and use various (possibly
> >> intersecting)
> >> >>>> > test groups with TestNG
> >> >>>> > * TestNG-style metadata is more convenient than JUnit
test suites
> >> >>>> (now
> >> >>>> > I agree with this statement). IMHO this is the main TestNG
> >> benefit.
> >> >>>> > * It is possible to run TestNG from command line
> >> >>>> > * There is also the special ant task for running TestNG
> >> >>>> > * Not everything can be configured with the ant task or
> >> command-line
> >> >>>> > params, sometimes extra test suite definition file
> >> "testng.xml" is
> >> >>>> > needed
> >> >>>> > * It is possible to run jUnit tests with TestNG ("testng.xml"
is
> >> >>>> > needed in this case)
> >> >>>> > * It is possible to run junit tests we currently have
in Harmony
> >> >>>> with
> >> >>>> > TestNG without any problems and modifications of the source
code.
> >> >>>> > However, we probably should write some number of TestNG
test
> >> suite
> >> >>>> > definition files "testng.xml" to be able to run all our
junit
> >> >>>> tests (I
> >> >>>> > have tried tests for bean module and some tests for luni)
> >> >>>> > * We can mix jUnit tests and TestNG tests in the single
test
> >> suite
> >> >>>> > configuration – i.e. single testng.xml file. We can
add TestNG
> >> >>>> > metadata to some test classes and leave other test classes
> >> untouched
> >> >>>> > * TestNG generates HTML reports in its own style, not
a very
> >> >>>> > convenient one IMHO
> >> >>>> > * It is also possible to generate JUnitReports from the
output
> >> >>>> > generated by TestNG
> >> >>>> > * Such reports will have a little bit different structure
since
> >> >>>> TestNG
> >> >>>> > doesn't provider any information about enclosing type
for test
> >> >>>> > methods. Names for tests (replacement for JUnit "test
> >> classes") and
> >> >>>> > test suites should be externally configured in "testng.xml"
> >> >>>> > * TestNG for Java 5 doesn't work on Harmony because some
> >> necessary
> >> >>>> > classes from java.util.concurrent package are missing
and it
> >> seems
> >> >>>> > that jsr14 target (we are currently using) doesn't support
> >> >>>> annotations
> >> >>>> > * TestNG for Java 1.4 (javadoc version) currently works
on
> >> Harmony
> >> >>>> > * I have half-way done script that converts TestNG 1.4
metadata
> >> >>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations) tests.
> >> >>>> >
> >> >>>>
> >> >>>> Excellent summary! Thanks a lot
> >> >>>>
> >> >>>> > The question I'd like to raise now is – aren't we ready
for
> >> TestNG
> >> >>>> > right now?
> >> >>>> I suppose we will use Java 5.0 annotations of TestNG, so it
seems
> >> >>>> now we
> >> >>>> are not ready for TestNG. But we can continue our feasibility
> >> study,
> >> >>>> just like what you have done, to know if TestNG really meets
our
> >> >>>> requirements or if there are any potential problems. Maybe
we could
> >> >>>> list
> >> >>>> a prerequisite list. e.g,
> >> >>>> 1) Harmony can fully self-host TestNG with Java5 annotations
> >> >>>> 2) Test groups are well-defined and agreed in community
> >> >>>> 3) Guidelines to write TestNG testcases
> >> >>>> 4) Take one module to run a pilot case
> >> >>>> ....
> >> >>>>
> >> >>>> Please correct me if I'm wrong
> >> >>>>
> >> >>>> > For example, we could replace our harness from jUnit to
> >> >>>> > TestNG and lazily start converting of some failing and
platform
> >> >>>> > dependent tests to javadoc version of TestNG. The rest
of the
> >> tests
> >> >>>> > will remain jUnit in fact. And when our VM will be ready
to
> >> handle
> >> >>>> > annotations we can convert all our TestNG 1.4 tests to
TestNG
> >> 1.5. I
> >> >>>> > understand that this idea may seem to be too early. But
anyway we
> >> >>>> will
> >> >>>> > need to change things some day since many people are unhappy
with
> >> >>>> the
> >> >>>> > current testing infrastructure (me for example).
> >> >>>> Not sure if we really want to involve another migration: TestNG
> >> >>>> javadoc
> >> >>>> -> TestNG annotation. Any comments?
> >> >>>>
> >> >>>> >
> >> >>>> > Thought? Suggestions? Opposite opinions?
> >> >>>> >
> >> >>>> > With Best Regards,
> >> >>>> >
> >> >>>> >
> >> >>>>
> >> >>>> --
> >> >>>> Richard Liang
> >> >>>> China Software Development Lab, IBM
> >
> >> --
> >> Richard Liang
> >> China Software Development Lab, IBM
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message