harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexei Zakharov" <alexei.zakha...@gmail.com>
Subject Re: [testing] metadata approach
Date Mon, 14 Aug 2006 11:53:20 GMT
Hi Oliver,

> But is j.u.c actually required to be in the runtime under test? I was
> thinking
> that j.u.c was only required for the VM actually running the harness,
> and all
> that gets run on the VM under test is the actual test method. If this
> was true,
> then we could run TestNG with the RI (which has j.u.c) and use the
> jvm option to specify the Harmony VM (which would not need j.u.c).

I afraid we cannot do like that. At least I was not successful last
time I tried to run tests using the jvm="<harmony>" option. As far as
I understand TestNG requires j.u.c for running every single test
method  because parallel running can be specified on the method level.
I mean in TestNG you can write something like this:
@Test(threadPoolSize = 7, invocationCount = 29)
This means that this method should be invoked from different threads.
And it seems that TestNG needs j.u.c to implement multithreading.


> Yes agreed, it is good to make group membership explicit as it facilitates
> inclusion/exclusion of groups, and makes it obvious which group tests
> belong to. Perhaps the same should be done for api tests, so we have a
> type.api group?

So you suggest to add @Test (groups={os.any, type.api}) to every api
test (that runs on every platform) without any defaults at all?

> I thought I had sent a mail out on this in the original thread, but I guess
> I never did (unless Thunderbird is hiding mail from me again!).

Just checked - there is no such mail in my gmail box, at least in the
"[classlib] Testing conventions - a proposal" thread.

> So, for example, if we were on a Windows x86 32bit machine, the Ant
> scripts would run test groups "os.shared", "os.windows", "os.windows.x86"
> and (if we ever get any 32/64bit specific tests) "os.windows.x86.32bit",
> or something similar.

Well, I like it in general. The only thing I still feel uncomfortable
with is "os.shared". When some code is shared among different
platforms it makes sense. But here we have a test shared by several
OSes. Does this sound natural? But I don't feel strongly about that
and will not object if everybody likes this.

With Best Regards,

2006/8/14, Oliver Deakin <oliver.deakin@googlemail.com>:
> Alexei Zakharov wrote:
> > Hi Oliver,
> >
> >> So perhaps the build system should be changed temporarily so that
> >> we dont self host the test harness? i.e. until we get
> >> java.util.concurrent,
> >> run Ant and the subsequent TestNG process with RI or other non-Harmony
> >> VM, and launch the tests with Harmony VM using the jvm option.
> >
> > The bad news is that TestNG requires j.u.c stuff even for single test
> > execution (i.e. in any case if jvm="<path to harmony>"). :( So if we
> > want to run annotated tests with TestNG (even from the command line)
> > we need j.u.c.
>
> But is j.u.c actually required to be in the runtime under test? I was
> thinking
> that j.u.c was only required for the VM actually running the harness,
> and all
> that gets run on the VM under test is the actual test method. If this
> was true,
> then we could run TestNG with the RI (which has j.u.c) and use the
> jvm option to specify the Harmony VM (which would not need j.u.c).
> Please correct me if Im wrong - I'm just trying to explore our options in
> the hope that we can start to move to TestNG soon.
>
> >
> >> That's odd - Thread.class in luni-kernel (VME v4) definitely contains a
> >> getId() method.
> >
> > May be I did something wrong - I will check tomorrow.
> >
> >> I do like them - in fact I was going to link his mail after that point
> >> but couldn't find it.
> >
> > Here is the link:
> > http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44B75E41.7020901@googlemail.com%3e
> >
>
> Thanks - it was good to reread.
>
> >
> > As far as I remember there was additions to the George's list like
> > using @Test (groups={"os.any"} ) rather than simple @Test for API
> > tests that run on any platform.
> >
>
> Yes agreed, it is good to make group membership explicit as it facilitates
> inclusion/exclusion of groups, and makes it obvious which group tests
> belong to. Perhaps the same should be done for api tests, so we have a
> type.api group?
> I thought I had sent a mail out on this in the original thread, but I guess
> I never did (unless Thunderbird is hiding mail from me again!).
>
> We have had discussions on the list about platform specific native
> code organisation, and I think organisation of platform specific tests
> can be handled in a similar way.
> So we would have an "os.shared" (or os.any, Im just going with shared
> as that is what we use for the native code) group that would run on all
> platforms. If there are platform specific tests, they would be grouped
> by os, then architecture (and then possibly word size).
> So, for example, if we were on a Windows x86 32bit machine, the Ant
> scripts would run test groups "os.shared", "os.windows", "os.windows.x86"
> and (if we ever get any 32/64bit specific tests) "os.windows.x86.32bit",
> or something similar.
>
> I think this tallies with what George was suggesting, and makes sense to
> me. Are there any objections to this approach?
>
> Regards,
> Oliver
>
> >> I really mean that we should make sure that everyone is happy with
> >> a certain set of group names before going ahead and applying them.
> >> Once they are decided upon, they should be added to the testing
> >> conventions webpage.
> >
> > Yes, agree.
> >
> >
> > With Best Regards,
> >
> > 2006/8/10, Oliver Deakin <oliver.deakin@googlemail.com>:
> >> Alexei Zakharov wrote:
> >> >> > We now have this, so let the TestNG debate continue :)
> >> >> Unfortunately, we still need java.util.concurrent :-(
> >> >
> >> > Yeah, TestNG 5.0 still throws java.lang.NoClassDefFoundError :
> >> > java.util.concurrent.LinkedBlockingQueue on Harmony+j9v4.
> >>
> >> So perhaps the build system should be changed temporarily so that
> >> we dont self host the test harness? i.e. until we get
> >> java.util.concurrent,
> >> run Ant and the subsequent TestNG process with RI or other non-Harmony
> >> VM, and launch the tests with Harmony VM using the jvm option.
> >> At least it will allow us to move forward with TestNG (if that's what we
> >> decide) without having to wait for java.util.concurrent. Then when
> >> we have j.u.c, start self-hosting again.
> >> Comments?
> >>
> >> >
> >> > I've also got an error while trying to compile TestNG 5.0 tests with
> >> > Harmony+j9v4+ecj: The method getId() is undefined for the type Thread
> >>
> >> That's odd - Thread.class in luni-kernel (VME v4) definitely contains a
> >> getId() method. I don't know anything about the TestNG tests - how are
> >> they run? is luni-kernel.jar definitely at the front of the
> >> bootclasspath?
> >>
> >> >
> >> >> > - If we go ahead with TestNG, we need to select a set of group
> >> names
> >> >> > to use to indicate exclusion, platform specificness etc.
> >> >
> >> > Don't you like the names suggested by George?
> >>
> >> I do like them - in fact I was going to link his mail after that point
> >> but couldn't
> >> find it. I really mean that we should make sure that everyone is
> >> happy with
> >> a certain set of group names before going ahead and applying them.
> >> Once they are decided upon, they should be added to the testing
> >> conventions webpage.
> >>
> >> >
> >> >> > - Decide whether some physical separation of tests on disk is
> >> >> necessary,  for instance to separate classpath and bootclasspath
> >> tests
> >> >
> >> > IMHO it is ok to separate classpath and bootclasspath tests because it
> >> > will be easer to implement such solution technically.
> >>
> >> I agree, although I don't feel strongly about it.
> >>
> >> Regards,
> >> Oliver
> >>
> >> >
> >> > Regards,
> >> >
> >> > 2006/8/10, Richard Liang <richard.liangyx@gmail.com>:
> >> >>
> >> >>
> >> >> Oliver Deakin wrote:
> >> >> > Richard Liang wrote:
> >> >> >>
> >> >> >>
> >> >> >> Alexei Zakharov wrote:
> >> >> >>> Hi Richard,
> >> >> >>>
> >> >> >>>> Not sure if we really want to involve another migration:
TestNG
> >> >> >>>> javadoc
> >> >> >>>> -> TestNG annotation. Any comments?
> >> >> >>>
> >> >> >>> Well, IMHO this depends on time constraints - when do
we plan to
> >> >> have
> >> >> >>> the support for anotations? If the answer is about a couple
of
> >> >> weeks -
> >> >> >>> no problem, we can wait. But if this is several months...
> >> >> >>> About the "migration" - I don't think this will be a real
> >> painfull
> >> >> >>> migration, all infrastructure will remain the same. We
will only
> >> >> need
> >> >> >>> to convert javadocs to annotations (one-one correspondence)
> >> and this
> >> >> >>> task can be easily automated.
> >> >> >> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell
> >> us when
> >> >> >> we will have a VM with annotation support?
> >> >> >
> >> >> > We now have this, so let the TestNG debate continue :)
> >> >> >
> >> >>
> >> >> Unfortunately, we still need java.util.concurrent :-(
> >> >>
> >> >>
> >> >> > I guess we need to decide a few things before we go ahead with
> >> this:
> >> >> > - Whether TestNG is generally accepted by the Harmony community
> >> >> > as our test harness of choice for unit testing. I think this will
> >> >> > probably
> >> >> > require a vote of some kind before we could make the move.
> >> >> > - If we go ahead with TestNG, we need to select a set of group
> >> names
> >> >> > to use
> >> >> > to indicate exclusion, platform specificness etc.
> >> >> > - Decide whether some physical separation of tests on disk is
> >> >> necessary,
> >> >> > for instance to separate classpath and bootclasspath tests.
> >> >> >
> >> >> > Comments/additions?
> >> >>
> >> >> Agree.  And we could provide proposals for these questions case by
> >> case,
> >> >> and let community make decision.
> >> >>
> >> >> Best regards,
> >> >> Richard
> >> >> >
> >> >> > Regards,
> >> >> > Oliver
> >> >> >
> >> >> >
> >> >> >>
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>>
> >> >> >>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Alexei Zakharov wrote:
> >> >> >>>> > Hi,
> >> >> >>>> >
> >> >> >>>> > I have created this new thread as a single place
for
> >> discussions
> >> >> >>>> > started in "Re: [testing] Peace" and "[classlib]
Testing
> >> >> >>>> conventions –
> >> >> >>>> > a proposal" threads.
> >> >> >>>> >
> >> >> >>>> > What did we have in the previous threads?
> >> >> >>>> > * Test classification proposed by Vladimir
> >> >> >>>> > * Test classification and group names proposed
by George
> >> >> >>>> > * Solution for Ant and TestNG scripting issues
(still being
> >> >> >>>> discussed)
> >> >> >>>> >
> >> >> >>>> > Since a lot of people are asking about TestNG
and wanting
> >> >> TestNG I
> >> >> >>>> > decide to put some effort and take a closer look
at this
> >> piece of
> >> >> >>>> > software. Thus during the last few days I was
playing with
> >> TestNG
> >> >> >>>> - I
> >> >> >>>> > tried to run different kind of tests with it,
to perform
> >> various
> >> >> >>>> > workloads, generate reports in different ways
and etc. The
> >> >> >>>> purpose of
> >> >> >>>> > all this activity was to try TestNG in a real
work,
> >> understand is
> >> >> >>>> > TestNG really worth our credits and how expensive
can be
> >> >> moving to
> >> >> >>>> > TestNG from our currently implemented testing
infrastructure.
> >> >> Now I
> >> >> >>>> > have some thoughts and facts I'd like to share
with the
> >> >> community.
> >> >> >>>> > I've put it in the form of list for convenience.
> >> >> >>>> >
> >> >> >>>> > * TestNG works ok in normal conditions, no visible
bugs
> >> >> >>>> > * It is possible to define and use various (possibly
> >> >> intersecting)
> >> >> >>>> > test groups with TestNG
> >> >> >>>> > * TestNG-style metadata is more convenient than
JUnit test
> >> suites
> >> >> >>>> (now
> >> >> >>>> > I agree with this statement). IMHO this is the
main TestNG
> >> >> benefit.
> >> >> >>>> > * It is possible to run TestNG from command line
> >> >> >>>> > * There is also the special ant task for running
TestNG
> >> >> >>>> > * Not everything can be configured with the ant
task or
> >> >> command-line
> >> >> >>>> > params, sometimes extra test suite definition
file
> >> >> "testng.xml" is
> >> >> >>>> > needed
> >> >> >>>> > * It is possible to run jUnit tests with TestNG
> >> ("testng.xml" is
> >> >> >>>> > needed in this case)
> >> >> >>>> > * It is possible to run junit tests we currently
have in
> >> Harmony
> >> >> >>>> with
> >> >> >>>> > TestNG without any problems and modifications
of the source
> >> code.
> >> >> >>>> > However, we probably should write some number
of TestNG test
> >> >> suite
> >> >> >>>> > definition files "testng.xml" to be able to run
all our junit
> >> >> >>>> tests (I
> >> >> >>>> > have tried tests for bean module and some tests
for luni)
> >> >> >>>> > * We can mix jUnit tests and TestNG tests in
the single test
> >> >> suite
> >> >> >>>> > configuration – i.e. single testng.xml file.
We can add TestNG
> >> >> >>>> > metadata to some test classes and leave other
test classes
> >> >> untouched
> >> >> >>>> > * TestNG generates HTML reports in its own style,
not a very
> >> >> >>>> > convenient one IMHO
> >> >> >>>> > * It is also possible to generate JUnitReports
from the output
> >> >> >>>> > generated by TestNG
> >> >> >>>> > * Such reports will have a little bit different
structure
> >> since
> >> >> >>>> TestNG
> >> >> >>>> > doesn't provider any information about enclosing
type for test
> >> >> >>>> > methods. Names for tests (replacement for JUnit
"test
> >> >> classes") and
> >> >> >>>> > test suites should be externally configured in
"testng.xml"
> >> >> >>>> > * TestNG for Java 5 doesn't work on Harmony because
some
> >> >> necessary
> >> >> >>>> > classes from java.util.concurrent package are
missing and it
> >> >> seems
> >> >> >>>> > that jsr14 target (we are currently using) doesn't
support
> >> >> >>>> annotations
> >> >> >>>> > * TestNG for Java 1.4 (javadoc version) currently
works on
> >> >> Harmony
> >> >> >>>> > * I have half-way done script that converts TestNG
1.4
> >> metadata
> >> >> >>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations)
tests.
> >> >> >>>> >
> >> >> >>>>
> >> >> >>>> Excellent summary! Thanks a lot
> >> >> >>>>
> >> >> >>>> > The question I'd like to raise now is – aren't
we ready for
> >> >> TestNG
> >> >> >>>> > right now?
> >> >> >>>> I suppose we will use Java 5.0 annotations of TestNG,
so it
> >> seems
> >> >> >>>> now we
> >> >> >>>> are not ready for TestNG. But we can continue our
feasibility
> >> >> study,
> >> >> >>>> just like what you have done, to know if TestNG really
meets our
> >> >> >>>> requirements or if there are any potential problems.
Maybe we
> >> could
> >> >> >>>> list
> >> >> >>>> a prerequisite list. e.g,
> >> >> >>>> 1) Harmony can fully self-host TestNG with Java5 annotations
> >> >> >>>> 2) Test groups are well-defined and agreed in community
> >> >> >>>> 3) Guidelines to write TestNG testcases
> >> >> >>>> 4) Take one module to run a pilot case
> >> >> >>>> ....
> >> >> >>>>
> >> >> >>>> Please correct me if I'm wrong
> >> >> >>>>
> >> >> >>>> > For example, we could replace our harness from
jUnit to
> >> >> >>>> > TestNG and lazily start converting of some failing
and
> >> platform
> >> >> >>>> > dependent tests to javadoc version of TestNG.
The rest of the
> >> >> tests
> >> >> >>>> > will remain jUnit in fact. And when our VM will
be ready to
> >> >> handle
> >> >> >>>> > annotations we can convert all our TestNG 1.4
tests to TestNG
> >> >> 1.5. I
> >> >> >>>> > understand that this idea may seem to be too
early. But
> >> anyway we
> >> >> >>>> will
> >> >> >>>> > need to change things some day since many people
are
> >> unhappy with
> >> >> >>>> the
> >> >> >>>> > current testing infrastructure (me for example).
> >> >> >>>> Not sure if we really want to involve another migration:
TestNG
> >> >> >>>> javadoc
> >> >> >>>> -> TestNG annotation. Any comments?
> >> >> >>>>
> >> >> >>>> >
> >> >> >>>> > Thought? Suggestions? Opposite opinions?
> >> >> >>>> >
> >> >> >>>> > With Best Regards,
> >> >> >>>> >
> >> >> >>>> >
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Richard Liang
> >> >> >>>> China Software Development Lab, IBM
> >> >
> >> >> --
> >> >> Richard Liang
> >> >> China Software Development Lab, IBM
> >> >
> >>
> >> --
> >> Oliver Deakin
> >> IBM United Kingdom Limited
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>


-- 
Alexei Zakharov,
Intel Middleware Product Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message