Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (asf.osuosl.org: 202.81.18.152 is neither permitted nor
 denied by domain of richard.liangyx@gmail.com)
Message-ID: <44DB0AF5.5020406@gmail.com>
Date: Thu, 10 Aug 2006 18:31:17 +0800
From: Richard Liang <richard.liangyx@gmail.com>
User-Agent: Thunderbird 1.5.0.5 (Windows/20060719)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [testing] metadata approach
References: <2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com>
	 <44CF0FEB.5050404@gmail.com>
 <2c9597b90608010430r5424c270mcbd19cdcb1570ae5@mail.gmail.com>
 <44D00B21.3030807@gmail.com> <44DB05EA.6030207@googlemail.com>
In-Reply-To: <44DB05EA.6030207@googlemail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: quoted-printable


Oliver Deakin wrote:
> Richard Liang wrote:
>>
>>
>> Alexei Zakharov wrote:
>>> Hi Richard,
>>>
>>>> Not sure if we really want to involve another migration: TestNG=20
>>>> javadoc
>>>> -> TestNG annotation. Any comments?
>>>
>>> Well, IMHO this depends on time constraints - when do we plan to have
>>> the support for anotations? If the answer is about a couple of weeks -
>>> no problem, we can wait. But if this is several months...
>>> About the "migration" - I don't think this will be a real painfull
>>> migration, all infrastructure will remain the same. We will only need
>>> to convert javadocs to annotations (one-one correspondence) and this
>>> task can be easily automated.
>> Sounds reasonable. :-)  Maybe drlvm guys or Oliver could tell us when=20
>> we will have a VM with annotation support?
>
> We now have this, so let the TestNG debate continue :)
>

Unfortunately, we still need java.util.concurrent :-(


> I guess we need to decide a few things before we go ahead with this:
> - Whether TestNG is generally accepted by the Harmony community
> as our test harness of choice for unit testing. I think this will=20
> probably
> require a vote of some kind before we could make the move.
> - If we go ahead with TestNG, we need to select a set of group names=20
> to use
> to indicate exclusion, platform specificness etc.
> - Decide whether some physical separation of tests on disk is necessary,
> for instance to separate classpath and bootclasspath tests.
>
> Comments/additions?=20

Agree.  And we could provide proposals for these questions case by case,=20
and let community make decision.

Best regards,
Richard
>
> Regards,
> Oliver
>
>
>>
>>>
>>> Thanks,
>>>
>>> 2006/8/1, Richard Liang <richard.liangyx@gmail.com>:
>>>>
>>>>
>>>> Alexei Zakharov wrote:
>>>> > Hi,
>>>> >
>>>> > I have created this new thread as a single place for discussions
>>>> > started in "Re: [testing] Peace" and "[classlib] Testing=20
>>>> conventions =96
>>>> > a proposal" threads.
>>>> >
>>>> > What did we have in the previous threads?
>>>> > * Test classification proposed by Vladimir
>>>> > * Test classification and group names proposed by George
>>>> > * Solution for Ant and TestNG scripting issues (still being=20
>>>> discussed)
>>>> >
>>>> > Since a lot of people are asking about TestNG and wanting TestNG I
>>>> > decide to put some effort and take a closer look at this piece of
>>>> > software. Thus during the last few days I was playing with TestNG=20
>>>> - I
>>>> > tried to run different kind of tests with it, to perform various
>>>> > workloads, generate reports in different ways and etc. The=20
>>>> purpose of
>>>> > all this activity was to try TestNG in a real work, understand is
>>>> > TestNG really worth our credits and how expensive can be moving to
>>>> > TestNG from our currently implemented testing infrastructure. Now I
>>>> > have some thoughts and facts I'd like to share with the community.
>>>> > I've put it in the form of list for convenience.
>>>> >
>>>> > * TestNG works ok in normal conditions, no visible bugs
>>>> > * It is possible to define and use various (possibly intersecting)
>>>> > test groups with TestNG
>>>> > * TestNG-style metadata is more convenient than JUnit test suites=20
>>>> (now
>>>> > I agree with this statement). IMHO this is the main TestNG benefit.
>>>> > * It is possible to run TestNG from command line
>>>> > * There is also the special ant task for running TestNG
>>>> > * Not everything can be configured with the ant task or command-line
>>>> > params, sometimes extra test suite definition file "testng.xml" is
>>>> > needed
>>>> > * It is possible to run jUnit tests with TestNG ("testng.xml" is
>>>> > needed in this case)
>>>> > * It is possible to run junit tests we currently have in Harmony=20
>>>> with
>>>> > TestNG without any problems and modifications of the source code.
>>>> > However, we probably should write some number of TestNG test suite
>>>> > definition files "testng.xml" to be able to run all our junit=20
>>>> tests (I
>>>> > have tried tests for bean module and some tests for luni)
>>>> > * We can mix jUnit tests and TestNG tests in the single test suite
>>>> > configuration =96 i.e. single testng.xml file. We can add TestNG
>>>> > metadata to some test classes and leave other test classes untouched
>>>> > * TestNG generates HTML reports in its own style, not a very
>>>> > convenient one IMHO
>>>> > * It is also possible to generate JUnitReports from the output
>>>> > generated by TestNG
>>>> > * Such reports will have a little bit different structure since=20
>>>> TestNG
>>>> > doesn't provider any information about enclosing type for test
>>>> > methods. Names for tests (replacement for JUnit "test classes") and
>>>> > test suites should be externally configured in "testng.xml"
>>>> > * TestNG for Java 5 doesn't work on Harmony because some necessary
>>>> > classes from java.util.concurrent package are missing and it seems
>>>> > that jsr14 target (we are currently using) doesn't support=20
>>>> annotations
>>>> > * TestNG for Java 1.4 (javadoc version) currently works on Harmony
>>>> > * I have half-way done script that converts TestNG 1.4 metadata
>>>> > (javadoc) tests to TestNG 1.5 (5.0 annotations) tests.
>>>> >
>>>>
>>>> Excellent summary! Thanks a lot
>>>>
>>>> > The question I'd like to raise now is =96 aren't we ready for TestNG
>>>> > right now?
>>>> I suppose we will use Java 5.0 annotations of TestNG, so it seems=20
>>>> now we
>>>> are not ready for TestNG. But we can continue our feasibility study,
>>>> just like what you have done, to know if TestNG really meets our
>>>> requirements or if there are any potential problems. Maybe we could=20
>>>> list
>>>> a prerequisite list. e.g,
>>>> 1) Harmony can fully self-host TestNG with Java5 annotations
>>>> 2) Test groups are well-defined and agreed in community
>>>> 3) Guidelines to write TestNG testcases
>>>> 4) Take one module to run a pilot case
>>>> ....
>>>>
>>>> Please correct me if I'm wrong
>>>>
>>>> > For example, we could replace our harness from jUnit to
>>>> > TestNG and lazily start converting of some failing and platform
>>>> > dependent tests to javadoc version of TestNG. The rest of the tests
>>>> > will remain jUnit in fact. And when our VM will be ready to handle
>>>> > annotations we can convert all our TestNG 1.4 tests to TestNG 1.5. I
>>>> > understand that this idea may seem to be too early. But anyway we=20
>>>> will
>>>> > need to change things some day since many people are unhappy with=20
>>>> the
>>>> > current testing infrastructure (me for example).
>>>> Not sure if we really want to involve another migration: TestNG=20
>>>> javadoc
>>>> -> TestNG annotation. Any comments?
>>>>
>>>> >
>>>> > Thought? Suggestions? Opposite opinions?
>>>> >
>>>> > With Best Regards,
>>>> >
>>>> >
>>>>
>>>> --=20
>>>> Richard Liang
>>>> China Software Development Lab, IBM
>>>
>>>
>>>
>>
>

--=20
Richard Liang
China Software Development Lab, IBM=20


---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org