Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: pass (asf.osuosl.org: domain of oliver.deakin@googlemail.com
 designates 64.233.182.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=googlemail.com;
        h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding;
        b=B9PzktzZ6YywSWelzIaxD23du5CBN13NhDwMWTk2fQiTeBdO0XbDUgHBUlM+F4W6Nue0x6usqE+SR3XweStMqUrILircDqP1jKFBNpEbLuZa4L2xZgN8c54eGaObUieJ4zeXnDWR21Ea37tlRWQ1329L8K8vP/1i86HXJgWofmM=
Message-ID: <44DB339A.4050108@googlemail.com>
Date: Thu, 10 Aug 2006 14:24:42 +0100
From: Oliver Deakin <oliver.deakin@googlemail.com>
User-Agent: Thunderbird 1.5.0.5 (Windows/20060719)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [testing] metadata approach
References: <2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com>
	 <44CF0FEB.5050404@gmail.com>
 <2c9597b90608010430r5424c270mcbd19cdcb1570ae5@mail.gmail.com>
 <44D00B21.3030807@gmail.com> <44DB05EA.6030207@googlemail.com>
 <44DB0AF5.5020406@gmail.com>
In-Reply-To: <44DB0AF5.5020406@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

Richard Liang wrote:
>
>
> Oliver Deakin wrote:
>> Richard Liang wrote:
>>>
>>>
>>> Alexei Zakharov wrote:
>>>> Hi Richard,
>>>>
>>>>> Not sure if we really want to involve another migration: TestNG 
>>>>> javadoc
>>>>> -> TestNG annotation. Any comments?
>>>>
>>>> Well, IMHO this depends on time constraints - when do we plan to have
>>>> the support for anotations? If the answer is about a couple of weeks -
>>>> no problem, we can wait. But if this is several months...
>>>> About the "migration" - I don't think this will be a real painfull
>>>> migration, all infrastructure will remain the same. We will only need
>>>> to convert javadocs to annotations (one-one correspondence) and this
>>>> task can be easily automated.
>>> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell us 
>>> when we will have a VM with annotation support?
>>
>> We now have this, so let the TestNG debate continue :)
>>
>
> Unfortunately, we still need java.util.concurrent :-(

Ah! I hadn't realised that that was a requirement of TestNG.

>
>
>> I guess we need to decide a few things before we go ahead with this:
>> - Whether TestNG is generally accepted by the Harmony community
>> as our test harness of choice for unit testing. I think this will 
>> probably
>> require a vote of some kind before we could make the move.
>> - If we go ahead with TestNG, we need to select a set of group names 
>> to use
>> to indicate exclusion, platform specificness etc.
>> - Decide whether some physical separation of tests on disk is necessary,
>> for instance to separate classpath and bootclasspath tests.
>>
>> Comments/additions? 
>
> Agree.  And we could provide proposals for these questions case by 
> case, and let community make decision.

Sounds good.

Regards,
Oliver

>
> Best regards,
> Richard
>>
>> Regards,
>> Oliver
>>
>>
>>>
>>>>
>>>> Thanks,
>>>>
>>>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
>>>>>
>>>>>
>>>>> Alexei Zakharov wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I have created this new thread as a single place for discussions
>>>>> > started in "Re: [testing] Peace" and "[classlib] Testing 
>>>>> conventions �
>>>>> > a proposal" threads.
>>>>> >
>>>>> > What did we have in the previous threads?
>>>>> > * Test classification proposed by Vladimir
>>>>> > * Test classification and group names proposed by George
>>>>> > * Solution for Ant and TestNG scripting issues (still being 
>>>>> discussed)
>>>>> >
>>>>> > Since a lot of people are asking about TestNG and wanting TestNG I
>>>>> > decide to put some effort and take a closer look at this piece of
>>>>> > software. Thus during the last few days I was playing with 
>>>>> TestNG - I
>>>>> > tried to run different kind of tests with it, to perform various
>>>>> > workloads, generate reports in different ways and etc. The 
>>>>> purpose of
>>>>> > all this activity was to try TestNG in a real work, understand is
>>>>> > TestNG really worth our credits and how expensive can be moving to
>>>>> > TestNG from our currently implemented testing infrastructure. Now I
>>>>> > have some thoughts and facts I'd like to share with the community.
>>>>> > I've put it in the form of list for convenience.
>>>>> >
>>>>> > * TestNG works ok in normal conditions, no visible bugs
>>>>> > * It is possible to define and use various (possibly intersecting)
>>>>> > test groups with TestNG
>>>>> > * TestNG-style metadata is more convenient than JUnit test 
>>>>> suites (now
>>>>> > I agree with this statement). IMHO this is the main TestNG benefit.
>>>>> > * It is possible to run TestNG from command line
>>>>> > * There is also the special ant task for running TestNG
>>>>> > * Not everything can be configured with the ant task or 
>>>>> command-line
>>>>> > params, sometimes extra test suite definition file "testng.xml" is
>>>>> > needed
>>>>> > * It is possible to run jUnit tests with TestNG ("testng.xml" is
>>>>> > needed in this case)
>>>>> > * It is possible to run junit tests we currently have in Harmony 
>>>>> with
>>>>> > TestNG without any problems and modifications of the source code.
>>>>> > However, we probably should write some number of TestNG test suite
>>>>> > definition files "testng.xml" to be able to run all our junit 
>>>>> tests (I
>>>>> > have tried tests for bean module and some tests for luni)
>>>>> > * We can mix jUnit tests and TestNG tests in the single test suite
>>>>> > configuration � i.e. single testng.xml file. We can add TestNG
>>>>> > metadata to some test classes and leave other test classes 
>>>>> untouched
>>>>> > * TestNG generates HTML reports in its own style, not a very
>>>>> > convenient one IMHO
>>>>> > * It is also possible to generate JUnitReports from the output
>>>>> > generated by TestNG
>>>>> > * Such reports will have a little bit different structure since 
>>>>> TestNG
>>>>> > doesn't provider any information about enclosing type for test
>>>>> > methods. Names for tests (replacement for JUnit "test classes") and
>>>>> > test suites should be externally configured in "testng.xml"
>>>>> > * TestNG for Java 5 doesn't work on Harmony because some necessary
>>>>> > classes from java.util.concurrent package are missing and it seems
>>>>> > that jsr14 target (we are currently using) doesn't support 
>>>>> annotations
>>>>> > * TestNG for Java 1.4 (javadoc version) currently works on Harmony
>>>>> > * I have half-way done script that converts TestNG 1.4 metadata
>>>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations) tests.
>>>>> >
>>>>>
>>>>> Excellent summary! Thanks a lot
>>>>>
>>>>> > The question I'd like to raise now is � aren't we ready for TestNG
>>>>> > right now?
>>>>> I suppose we will use Java 5.0 annotations of TestNG, so it seems 
>>>>> now we
>>>>> are not ready for TestNG. But we can continue our feasibility study,
>>>>> just like what you have done, to know if TestNG really meets our
>>>>> requirements or if there are any potential problems. Maybe we 
>>>>> could list
>>>>> a prerequisite list. e.g,
>>>>> 1) Harmony can fully self-host TestNG with Java5 annotations
>>>>> 2) Test groups are well-defined and agreed in community
>>>>> 3) Guidelines to write TestNG testcases
>>>>> 4) Take one module to run a pilot case
>>>>> ....
>>>>>
>>>>> Please correct me if I'm wrong
>>>>>
>>>>> > For example, we could replace our harness from jUnit to
>>>>> > TestNG and lazily start converting of some failing and platform
>>>>> > dependent tests to javadoc version of TestNG. The rest of the tests
>>>>> > will remain jUnit in fact. And when our VM will be ready to handle
>>>>> > annotations we can convert all our TestNG 1.4 tests to TestNG 
>>>>> 1.5. I
>>>>> > understand that this idea may seem to be too early. But anyway 
>>>>> we will
>>>>> > need to change things some day since many people are unhappy 
>>>>> with the
>>>>> > current testing infrastructure (me for example).
>>>>> Not sure if we really want to involve another migration: TestNG 
>>>>> javadoc
>>>>> -> TestNG annotation. Any comments?
>>>>>
>>>>> >
>>>>> > Thought? Suggestions? Opposite opinions?
>>>>> >
>>>>> > With Best Regards,
>>>>> >
>>>>> >
>>>>>
>>>>> -- 
>>>>> Richard Liang
>>>>> China Software Development Lab, IBM
>>>>
>>>>
>>>>
>>>
>>
>

-- 
Oliver Deakin
IBM United Kingdom Limited


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org