harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [classlib][test] Migration to testNG?
Date Thu, 12 Jun 2008 07:08:14 GMT
By groups  annotation, we can group them by arch.<arch id>, type.impl,
os.<os id>, vm.<vm id>.
These groups indicate that we have different behaviors but it is allowed.
Again, we can keep track of them without moving them to other
directories by TestNG.

2008/6/12 Nathan Beyer <ndbeyer@apache.org>:
> I think having a mechanism for exclusions of failing tests is bad,
> especially any mechanism that allows excluding tests to sit hidden within
> working tests. I think tests that are excluded because they are failing and
> aren't going to be fixed need to be moved out of the running tests and
> dropped in an separate folder like "src/test/failing" so that they stand
> out.
>
> -Nathan
>
> On Wed, Jun 11, 2008 at 6:46 AM, Regis <xu.regis@gmail.com> wrote:
>
>>
>> Junit 4 has @ignore annotation which of target is method, which can ignore
>> failing methods.
>> But some modules, such as luni, have different exclude lists on different
>> os or vm, I haven't
>> found any ways to do this using @ignore annotation. While TestNG's @group
>> annotation
>> can do this easily, just define groups for different exclude lists.
>>
>> Best Regards,
>> Regis.
>>
>>
>> Alexei Zakharov wrote:
>>
>>> As far as I understand in spite of the fact there were no
>>> TestNG-related discussions since 2006 the problem is still relevant.
>>> There are big exclude lists in some classlib modules still, and many
>>> tests are excluded only because of a couple of failing methods.
>>> Frankly speaking I'm not familiar with new feature introduced in Junit
>>> 4.4. Are there any enhancements that can help to resolve this
>>> exclude-whole-class-because-of-one-bad-method issue?
>>>
>>> Thanks,
>>> Alexei
>>>
>>> 2008/6/11, Regis <xu.regis@gmail.com>:
>>>
>>>> Nathan Beyer wrote:
>>>>
>>>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu.regis@gmail.com> wrote:
>>>>>
>>>>>
>>>>>  Hi,
>>>>>>
>>>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>>>> They would be very helpful for our new test cases. But I notice that
>>>>>> Junit 4.4 doesn't support group which is very important feature for
>>>>>> both old tests and new tests. We can partition our test suite and
run
>>>>>> them separately. It's make our tests more flexible and configurable,
>>>>>> and it's the main reason we discuss to migrate to TestNG long time
ago.
>>>>>>
>>>>>>
>>>>> Don't we partition our tests already? Isn't that what the 'api' and
>>>>> 'impl'
>>>>> folders are about?
>>>>>
>>>>>  Yes, but it's not enough. We have discussed and created a wiki page[1]
>>>> about
>>>> how
>>>> to configuration and group harmony tests. The page is a little old, but I
>>>> think the problems
>>>> it tried to resolve still exist now. The partitions are not only include
>>>> 'api' and 'impl', but also
>>>> include partition of different os, architecture, partition of broken
>>>> tests
>>>> and level of tests.
>>>> folder structure or exclude files can't help in this complex situation,
>>>> so
>>>> we need some tools
>>>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
>>>> can
>>>> do it, i will vote to JUnit,
>>>> update to a new version is always easier than switch to a new tool after
>>>> all.
>>>>
>>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>>
>>>> Best Regards,
>>>> Regis.
>>>>
>>>>
>>>>  -Nathan
>>>>>
>>>>>
>>>>>
>>>>>  Best Regards,
>>>>>> Regis.
>>>>>>
>>>>>>
>>>>>> Nathan Beyer wrote:
>>>>>>
>>>>>>
>>>>>>  That discussion was a very long time ago. Is there still value in
>>>>>>>
>>>>>> TestNG?
>>>>
>>>>> I'd prefer to move to JUnit 4.4. All of our current tests will
>>>>>>>
>>>>>> continue to
>>>>
>>>>> work and new tests can be implemented using the latest conventions and
>>>>>>> older
>>>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry
from
>>>>>>>
>>>>>> 4.0.
>>>>
>>>>> Here's the things I think would be create for our use and testing in
>>>>>>> general
>>>>>>> - Matchers and the 'assertThat' - much more readable code and
readable
>>>>>>> failure messages
>>>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>>>>>
>>>>>> that
>>>>
>>>>> guarantee that preconditions for the test are correct; this allows
>>>>>>>
>>>>>> tests
>>>>
>>>>> to
>>>>>>> fail such that you know it's an environment issue and not an
actual
>>>>>>>
>>>>>> test
>>>>
>>>>> failure
>>>>>>>
>>>>>>> If you're not familiar with matchers, check out this quick tutorial
-
>>>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>>>
>>>>>>> -Nathan
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <sean.xx.qiu@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>>  Hi, all.
>>>>>>>
>>>>>>>  We had discussed the migration to testNG before and got some
>>>>>>>>
>>>>>>> conclusions
>>>>
>>>>> for
>>>>>>>> grouping[1]
>>>>>>>> including how to deal with boot path test[2]. Am i missing
>>>>>>>>
>>>>>>> something?
>>>>
>>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>>>> I volunteer to carry out this job if no one objects.  Any
other
>>>>>>>> volunteers?
>>>>>>>>
>>>>>>>> IMHO, we can only add some ant tasks to integrate testng
at the
>>>>>>>> beginning.
>>>>>>>> So our original junit tests can still work at the mean time
when
>>>>>>>> migrating.
>>>>>>>> When one module's migration task is finished, we can judge
the
>>>>>>>>
>>>>>>> result
>>>>
>>>>> to dertermine whether we should go on for other modules.
>>>>>>>>
>>>>>>>> Maybe we can create a branch for luni to start this work,
shall we?
>>>>>>>> therefore there won't be any impact on other's development.
>>>>>>>> Once it is completed in the branch, we could merge it back
to our
>>>>>>>>
>>>>>>> trunk.
>>>>
>>>>> Does it make sense?
>>>>>>>>
>>>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>> http://wiki.apache.org/harmony/Testing_Convention
>>>>
>>>>> [2]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>>
>>>>> [3]
>>>>>>>>
>>>>>>> http://testng.org/doc/documentation-main.html#annotations
>>>>
>>>>>  --
>>>>>>>> Best Regards
>>>>>>>> Sean, Xiao Xia Qiu
>>>>>>>>  China Software Development Lab, IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>
>



-- 
Best Regards
Sean, Xiao Xia Qiu

China Software Development Lab, IBM

Mime
View raw message