harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oliver Deakin <oliver.dea...@googlemail.com>
Subject Re: [testing] metadata approach
Date Thu, 10 Aug 2006 10:09:46 GMT
Richard Liang wrote:
>
>
> Alexei Zakharov wrote:
>> Hi Richard,
>>
>>> Not sure if we really want to involve another migration: TestNG javadoc
>>> -> TestNG annotation. Any comments?
>>
>> Well, IMHO this depends on time constraints - when do we plan to have
>> the support for anotations? If the answer is about a couple of weeks -
>> no problem, we can wait. But if this is several months...
>> About the "migration" - I don't think this will be a real painfull
>> migration, all infrastructure will remain the same. We will only need
>> to convert javadocs to annotations (one-one correspondence) and this
>> task can be easily automated.
> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell us when 
> we will have a VM with annotation support?

We now have this, so let the TestNG debate continue :)

I guess we need to decide a few things before we go ahead with this:
 - Whether TestNG is generally accepted by the Harmony community
as our test harness of choice for unit testing. I think this will probably
require a vote of some kind before we could make the move.
 - If we go ahead with TestNG, we need to select a set of group names to use
to indicate exclusion, platform specificness etc.
 - Decide whether some physical separation of tests on disk is necessary,
for instance to separate classpath and bootclasspath tests.

Comments/additions?

Regards,
Oliver


>
>>
>> Thanks,
>>
>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
>>>
>>>
>>> Alexei Zakharov wrote:
>>> > Hi,
>>> >
>>> > I have created this new thread as a single place for discussions
>>> > started in "Re: [testing] Peace" and "[classlib] Testing 
>>> conventions –
>>> > a proposal" threads.
>>> >
>>> > What did we have in the previous threads?
>>> > * Test classification proposed by Vladimir
>>> > * Test classification and group names proposed by George
>>> > * Solution for Ant and TestNG scripting issues (still being 
>>> discussed)
>>> >
>>> > Since a lot of people are asking about TestNG and wanting TestNG I
>>> > decide to put some effort and take a closer look at this piece of
>>> > software. Thus during the last few days I was playing with TestNG - I
>>> > tried to run different kind of tests with it, to perform various
>>> > workloads, generate reports in different ways and etc. The purpose of
>>> > all this activity was to try TestNG in a real work, understand is
>>> > TestNG really worth our credits and how expensive can be moving to
>>> > TestNG from our currently implemented testing infrastructure. Now I
>>> > have some thoughts and facts I'd like to share with the community.
>>> > I've put it in the form of list for convenience.
>>> >
>>> > * TestNG works ok in normal conditions, no visible bugs
>>> > * It is possible to define and use various (possibly intersecting)
>>> > test groups with TestNG
>>> > * TestNG-style metadata is more convenient than JUnit test suites 
>>> (now
>>> > I agree with this statement). IMHO this is the main TestNG benefit.
>>> > * It is possible to run TestNG from command line
>>> > * There is also the special ant task for running TestNG
>>> > * Not everything can be configured with the ant task or command-line
>>> > params, sometimes extra test suite definition file "testng.xml" is
>>> > needed
>>> > * It is possible to run jUnit tests with TestNG ("testng.xml" is
>>> > needed in this case)
>>> > * It is possible to run junit tests we currently have in Harmony with
>>> > TestNG without any problems and modifications of the source code.
>>> > However, we probably should write some number of TestNG test suite
>>> > definition files "testng.xml" to be able to run all our junit 
>>> tests (I
>>> > have tried tests for bean module and some tests for luni)
>>> > * We can mix jUnit tests and TestNG tests in the single test suite
>>> > configuration – i.e. single testng.xml file. We can add TestNG
>>> > metadata to some test classes and leave other test classes untouched
>>> > * TestNG generates HTML reports in its own style, not a very
>>> > convenient one IMHO
>>> > * It is also possible to generate JUnitReports from the output
>>> > generated by TestNG
>>> > * Such reports will have a little bit different structure since 
>>> TestNG
>>> > doesn't provider any information about enclosing type for test
>>> > methods. Names for tests (replacement for JUnit "test classes") and
>>> > test suites should be externally configured in "testng.xml"
>>> > * TestNG for Java 5 doesn't work on Harmony because some necessary
>>> > classes from java.util.concurrent package are missing and it seems
>>> > that jsr14 target (we are currently using) doesn't support 
>>> annotations
>>> > * TestNG for Java 1.4 (javadoc version) currently works on Harmony
>>> > * I have half-way done script that converts TestNG 1.4 metadata
>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations) tests.
>>> >
>>>
>>> Excellent summary! Thanks a lot
>>>
>>> > The question I'd like to raise now is – aren't we ready for TestNG
>>> > right now?
>>> I suppose we will use Java 5.0 annotations of TestNG, so it seems 
>>> now we
>>> are not ready for TestNG. But we can continue our feasibility study,
>>> just like what you have done, to know if TestNG really meets our
>>> requirements or if there are any potential problems. Maybe we could 
>>> list
>>> a prerequisite list. e.g,
>>> 1) Harmony can fully self-host TestNG with Java5 annotations
>>> 2) Test groups are well-defined and agreed in community
>>> 3) Guidelines to write TestNG testcases
>>> 4) Take one module to run a pilot case
>>> ....
>>>
>>> Please correct me if I'm wrong
>>>
>>> > For example, we could replace our harness from jUnit to
>>> > TestNG and lazily start converting of some failing and platform
>>> > dependent tests to javadoc version of TestNG. The rest of the tests
>>> > will remain jUnit in fact. And when our VM will be ready to handle
>>> > annotations we can convert all our TestNG 1.4 tests to TestNG 1.5. I
>>> > understand that this idea may seem to be too early. But anyway we 
>>> will
>>> > need to change things some day since many people are unhappy with the
>>> > current testing infrastructure (me for example).
>>> Not sure if we really want to involve another migration: TestNG javadoc
>>> -> TestNG annotation. Any comments?
>>>
>>> >
>>> > Thought? Suggestions? Opposite opinions?
>>> >
>>> > With Best Regards,
>>> >
>>> >
>>>
>>> -- 
>>> Richard Liang
>>> China Software Development Lab, IBM
>>
>>
>>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message