Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of oliver.deakin@googlemail.com
 designates 64.233.182.185 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=googlemail.com;
        h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
        b=C0nNdrsPAWaEp9mMFljgCWTItyhzsJL4WA191RVmyAHagb+gmsC4QrjrQDlQJ+svgbZjUcFDJ3PrIEcme9TIkC5iEIksQXP0j681E40XMh6lWOGxE/Yc5XHFCJCVm+iL3w9sqwFwUHtqd5NHjbqNuexPOaUCHdq3qu2P0G32PTQ=
Message-ID: <44DB05EA.6030207@googlemail.com>
Date: Thu, 10 Aug 2006 11:09:46 +0100
From: Oliver Deakin <oliver.deakin@googlemail.com>
User-Agent: Thunderbird 1.5.0.5 (Windows/20060719)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [testing] metadata approach
References: <2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com>
	 <44CF0FEB.5050404@gmail.com>
 <2c9597b90608010430r5424c270mcbd19cdcb1570ae5@mail.gmail.com>
 <44D00B21.3030807@gmail.com>
In-Reply-To: <44D00B21.3030807@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

Richard Liang wrote:
>
>
> Alexei Zakharov wrote:
>> Hi Richard,
>>
>>> Not sure if we really want to involve another migration: TestNG javadoc
>>> -> TestNG annotation. Any comments?
>>
>> Well, IMHO this depends on time constraints - when do we plan to have
>> the support for anotations? If the answer is about a couple of weeks -
>> no problem, we can wait. But if this is several months...
>> About the "migration" - I don't think this will be a real painfull
>> migration, all infrastructure will remain the same. We will only need
>> to convert javadocs to annotations (one-one correspondence) and this
>> task can be easily automated.
> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell us when 
> we will have a VM with annotation support?

We now have this, so let the TestNG debate continue :)

I guess we need to decide a few things before we go ahead with this:
 - Whether TestNG is generally accepted by the Harmony community
as our test harness of choice for unit testing. I think this will probably
require a vote of some kind before we could make the move.
 - If we go ahead with TestNG, we need to select a set of group names to use
to indicate exclusion, platform specificness etc.
 - Decide whether some physical separation of tests on disk is necessary,
for instance to separate classpath and bootclasspath tests.

Comments/additions?

Regards,
Oliver


>
>>
>> Thanks,
>>
>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
>>>
>>>
>>> Alexei Zakharov wrote:
>>> > Hi,
>>> >
>>> > I have created this new thread as a single place for discussions
>>> > started in "Re: [testing] Peace" and "[classlib] Testing 
>>> conventions �
>>> > a proposal" threads.
>>> >
>>> > What did we have in the previous threads?
>>> > * Test classification proposed by Vladimir
>>> > * Test classification and group names proposed by George
>>> > * Solution for Ant and TestNG scripting issues (still being 
>>> discussed)
>>> >
>>> > Since a lot of people are asking about TestNG and wanting TestNG I
>>> > decide to put some effort and take a closer look at this piece of
>>> > software. Thus during the last few days I was playing with TestNG - I
>>> > tried to run different kind of tests with it, to perform various
>>> > workloads, generate reports in different ways and etc. The purpose of
>>> > all this activity was to try TestNG in a real work, understand is
>>> > TestNG really worth our credits and how expensive can be moving to
>>> > TestNG from our currently implemented testing infrastructure. Now I
>>> > have some thoughts and facts I'd like to share with the community.
>>> > I've put it in the form of list for convenience.
>>> >
>>> > * TestNG works ok in normal conditions, no visible bugs
>>> > * It is possible to define and use various (possibly intersecting)
>>> > test groups with TestNG
>>> > * TestNG-style metadata is more convenient than JUnit test suites 
>>> (now
>>> > I agree with this statement). IMHO this is the main TestNG benefit.
>>> > * It is possible to run TestNG from command line
>>> > * There is also the special ant task for running TestNG
>>> > * Not everything can be configured with the ant task or command-line
>>> > params, sometimes extra test suite definition file "testng.xml" is
>>> > needed
>>> > * It is possible to run jUnit tests with TestNG ("testng.xml" is
>>> > needed in this case)
>>> > * It is possible to run junit tests we currently have in Harmony with
>>> > TestNG without any problems and modifications of the source code.
>>> > However, we probably should write some number of TestNG test suite
>>> > definition files "testng.xml" to be able to run all our junit 
>>> tests (I
>>> > have tried tests for bean module and some tests for luni)
>>> > * We can mix jUnit tests and TestNG tests in the single test suite
>>> > configuration � i.e. single testng.xml file. We can add TestNG
>>> > metadata to some test classes and leave other test classes untouched
>>> > * TestNG generates HTML reports in its own style, not a very
>>> > convenient one IMHO
>>> > * It is also possible to generate JUnitReports from the output
>>> > generated by TestNG
>>> > * Such reports will have a little bit different structure since 
>>> TestNG
>>> > doesn't provider any information about enclosing type for test
>>> > methods. Names for tests (replacement for JUnit "test classes") and
>>> > test suites should be externally configured in "testng.xml"
>>> > * TestNG for Java 5 doesn't work on Harmony because some necessary
>>> > classes from java.util.concurrent package are missing and it seems
>>> > that jsr14 target (we are currently using) doesn't support 
>>> annotations
>>> > * TestNG for Java 1.4 (javadoc version) currently works on Harmony
>>> > * I have half-way done script that converts TestNG 1.4 metadata
>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations) tests.
>>> >
>>>
>>> Excellent summary! Thanks a lot
>>>
>>> > The question I'd like to raise now is � aren't we ready for TestNG
>>> > right now?
>>> I suppose we will use Java 5.0 annotations of TestNG, so it seems 
>>> now we
>>> are not ready for TestNG. But we can continue our feasibility study,
>>> just like what you have done, to know if TestNG really meets our
>>> requirements or if there are any potential problems. Maybe we could 
>>> list
>>> a prerequisite list. e.g,
>>> 1) Harmony can fully self-host TestNG with Java5 annotations
>>> 2) Test groups are well-defined and agreed in community
>>> 3) Guidelines to write TestNG testcases
>>> 4) Take one module to run a pilot case
>>> ....
>>>
>>> Please correct me if I'm wrong
>>>
>>> > For example, we could replace our harness from jUnit to
>>> > TestNG and lazily start converting of some failing and platform
>>> > dependent tests to javadoc version of TestNG. The rest of the tests
>>> > will remain jUnit in fact. And when our VM will be ready to handle
>>> > annotations we can convert all our TestNG 1.4 tests to TestNG 1.5. I
>>> > understand that this idea may seem to be too early. But anyway we 
>>> will
>>> > need to change things some day since many people are unhappy with the
>>> > current testing infrastructure (me for example).
>>> Not sure if we really want to involve another migration: TestNG javadoc
>>> -> TestNG annotation. Any comments?
>>>
>>> >
>>> > Thought? Suggestions? Opposite opinions?
>>> >
>>> > With Best Regards,
>>> >
>>> >
>>>
>>> -- 
>>> Richard Liang
>>> China Software Development Lab, IBM
>>
>>
>>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org