harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [classlib][test] Migration to testNG?
Date Thu, 12 Jun 2008 05:48:53 GMT
Sorry for late response.

For your question, I am not familiar with Junit 4.4 too, but i know
TestNG could figure it out indeed.
 "groups" annotation of TestNG should be a great candidate to solve
this problem.
We defined  two groups of state.broken.<os> and state.broken<vm> in
our wiki page[1] in this case.
We can group all these failed methods by these annotations to exclude
them when running all tests.
Once they are fixed , just remove them from state.broken group to other groups.
We are not required to exclude whole test class for one failure of its method.
The complex excluding files are not desired  any longer, just "groups"
annotations instead.

By means of "groups" annotation, we can replace our current approach
which classify our test by directories[2].
It is complex and hard to maintain because we don't only have api and
impl tests,
but also platform-specific  tests, run on classpath tests,
run on bootclasspath, behaves different between Harmony and RI tests
and so on...
We can find that tests in different modules may divided in different
directory structures currently.
If we adopt TestNG, this nighmare might be gone. All the directories
could be replaced by "groups" annotation.
They can be placed in the same directory as Maven's convention.
So we can archive them into a single jar.( the bootclass path test may
be a exception).
We can just tell the groups parameter to run our desired tests.

IMHO, for these two reasons , TestNG is totally worthy to be considered.

BTW,  i have successfully run our luni tests after converting from
Junit to TestNG.
I am trying to replace these excluding files by TestNG "groups"
annotation locally now.


[1] http://wiki.apache.org/harmony/Testing_Convention
[2] http://harmony.apache.org/subcomponents/classlibrary/testing.html

2008/6/11 Alexei Zakharov <alexei.zakharov@gmail.com>:
> As far as I understand in spite of the fact there were no
> TestNG-related discussions since 2006 the problem is still relevant.
> There are big exclude lists in some classlib modules still, and many
> tests are excluded only because of a couple of failing methods.
> Frankly speaking I'm not familiar with new feature introduced in Junit
> 4.4. Are there any enhancements that can help to resolve this
> exclude-whole-class-because-of-one-bad-method issue?
>
> Thanks,
> Alexei
>
> 2008/6/11, Regis <xu.regis@gmail.com>:
>>
>> Nathan Beyer wrote:
>> > On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu.regis@gmail.com> wrote:
>> >
>> >
>> > > Hi,
>> > >
>> > > Matcher and Assumptions are great ideas! Thanks Nathan.
>> > > They would be very helpful for our new test cases. But I notice that
>> > > Junit 4.4 doesn't support group which is very important feature for
>> > > both old tests and new tests. We can partition our test suite and run
>> > > them separately. It's make our tests more flexible and configurable,
>> > > and it's the main reason we discuss to migrate to TestNG long time ago.
>> > >
>> >
>> >
>> > Don't we partition our tests already? Isn't that what the 'api' and 'impl'
>> > folders are about?
>> >
>> Yes, but it's not enough. We have discussed and created a wiki page[1] about
>> how
>> to configuration and group harmony tests. The page is a little old, but I
>> think the problems
>> it tried to resolve still exist now. The partitions are not only include
>> 'api' and 'impl', but also
>> include partition of different os, architecture, partition of broken tests
>> and level of tests.
>> folder structure or exclude files can't help in this complex situation, so
>> we need some tools
>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
>> do it, i will vote to JUnit,
>> update to a new version is always easier than switch to a new tool after
>> all.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>
>> Best Regards,
>> Regis.
>>
>>
>> >
>> > -Nathan
>> >
>> >
>> >
>> > >
>> > > Best Regards,
>> > > Regis.
>> > >
>> > >
>> > > Nathan Beyer wrote:
>> > >
>> > >
>> > > > That discussion was a very long time ago. Is there still value in
>> TestNG?
>> > > > I'd prefer to move to JUnit 4.4. All of our current tests will
>> continue to
>> > > > work and new tests can be implemented using the latest conventions
and
>> > > > older
>> > > > tests can be updated as we get to them. JUnit 4.4 is a far cry from
>> 4.0.
>> > > >
>> > > > Here's the things I think would be create for our use and testing
in
>> > > > general
>> > > > - Matchers and the 'assertThat' - much more readable code and readable
>> > > > failure messages
>> > > > - Assumptions and the 'assumeThat' - allows methods to add statements
>> that
>> > > > guarantee that preconditions for the test are correct; this allows
>> tests
>> > > > to
>> > > > fail such that you know it's an environment issue and not an actual
>> test
>> > > > failure
>> > > >
>> > > > If you're not familiar with matchers, check out this quick tutorial
-
>> > > > http://code.google.com/p/hamcrest/wiki/Tutorial.
>> > > >
>> > > > -Nathan
>> > > >
>> > > >
>> > > > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <sean.xx.qiu@gmail.com>
>> wrote:
>> > > >
>> > > >  Hi, all.
>> > > >
>> > > > > We had discussed the migration to testNG before and got some
>> conclusions
>> > > > > for
>> > > > > grouping[1]
>> > > > > including how to deal with boot path test[2]. Am i missing
>> something?
>> > > > > Is it still in our schedule? I think it's valueable to Harmony.
>> > > > > I volunteer to carry out this job if no one objects.  Any other
>> > > > > volunteers?
>> > > > >
>> > > > > IMHO, we can only add some ant tasks to integrate testng at the
>> > > > > beginning.
>> > > > > So our original junit tests can still work at the mean time when
>> > > > > migrating.
>> > > > > When one module's migration task is finished, we can judge the
>> result
>> > > > > to dertermine whether we should go on for other modules.
>> > > > >
>> > > > > Maybe we can create a branch for luni to start this work, shall
we?
>> > > > > therefore there won't be any impact on other's development.
>> > > > > Once it is completed in the branch, we could merge it back to
our
>> trunk.
>> > > > > Does it make sense?
>> > > > >
>> > > > > Any sugestions or comments are welcomed. Thanks very much.
>> > > > >
>> > > > > [1]
>> http://wiki.apache.org/harmony/Testing_Convention
>> > > > > [2]
>> > > > >
>> > > > >
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>> > > > > [3]
>> http://testng.org/doc/documentation-main.html#annotations
>> > > > >  --
>> > > > > Best Regards
>> > > > > Sean, Xiao Xia Qiu
>> > > > >  China Software Development Lab, IBM
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>>
>



-- 
Best Regards
Sean, Xiao Xia Qiu

China Software Development Lab, IBM

Mime
View raw message