harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Beyer" <ndbe...@apache.org>
Subject Re: [classlib][test] Migration to testNG?
Date Thu, 12 Jun 2008 01:25:32 GMT
I think having a mechanism for exclusions of failing tests is bad,
especially any mechanism that allows excluding tests to sit hidden within
working tests. I think tests that are excluded because they are failing and
aren't going to be fixed need to be moved out of the running tests and
dropped in an separate folder like "src/test/failing" so that they stand
out.

-Nathan

On Wed, Jun 11, 2008 at 6:46 AM, Regis <xu.regis@gmail.com> wrote:

>
> Junit 4 has @ignore annotation which of target is method, which can ignore
> failing methods.
> But some modules, such as luni, have different exclude lists on different
> os or vm, I haven't
> found any ways to do this using @ignore annotation. While TestNG's @group
> annotation
> can do this easily, just define groups for different exclude lists.
>
> Best Regards,
> Regis.
>
>
> Alexei Zakharov wrote:
>
>> As far as I understand in spite of the fact there were no
>> TestNG-related discussions since 2006 the problem is still relevant.
>> There are big exclude lists in some classlib modules still, and many
>> tests are excluded only because of a couple of failing methods.
>> Frankly speaking I'm not familiar with new feature introduced in Junit
>> 4.4. Are there any enhancements that can help to resolve this
>> exclude-whole-class-because-of-one-bad-method issue?
>>
>> Thanks,
>> Alexei
>>
>> 2008/6/11, Regis <xu.regis@gmail.com>:
>>
>>> Nathan Beyer wrote:
>>>
>>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu.regis@gmail.com> wrote:
>>>>
>>>>
>>>>  Hi,
>>>>>
>>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>>> They would be very helpful for our new test cases. But I notice that
>>>>> Junit 4.4 doesn't support group which is very important feature for
>>>>> both old tests and new tests. We can partition our test suite and run
>>>>> them separately. It's make our tests more flexible and configurable,
>>>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>>>
>>>>>
>>>> Don't we partition our tests already? Isn't that what the 'api' and
>>>> 'impl'
>>>> folders are about?
>>>>
>>>>  Yes, but it's not enough. We have discussed and created a wiki page[1]
>>> about
>>> how
>>> to configuration and group harmony tests. The page is a little old, but I
>>> think the problems
>>> it tried to resolve still exist now. The partitions are not only include
>>> 'api' and 'impl', but also
>>> include partition of different os, architecture, partition of broken
>>> tests
>>> and level of tests.
>>> folder structure or exclude files can't help in this complex situation,
>>> so
>>> we need some tools
>>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
>>> can
>>> do it, i will vote to JUnit,
>>> update to a new version is always easier than switch to a new tool after
>>> all.
>>>
>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>
>>> Best Regards,
>>> Regis.
>>>
>>>
>>>  -Nathan
>>>>
>>>>
>>>>
>>>>  Best Regards,
>>>>> Regis.
>>>>>
>>>>>
>>>>> Nathan Beyer wrote:
>>>>>
>>>>>
>>>>>  That discussion was a very long time ago. Is there still value in
>>>>>>
>>>>> TestNG?
>>>
>>>> I'd prefer to move to JUnit 4.4. All of our current tests will
>>>>>>
>>>>> continue to
>>>
>>>> work and new tests can be implemented using the latest conventions and
>>>>>> older
>>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from
>>>>>>
>>>>> 4.0.
>>>
>>>> Here's the things I think would be create for our use and testing in
>>>>>> general
>>>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>>>> failure messages
>>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>>>>
>>>>> that
>>>
>>>> guarantee that preconditions for the test are correct; this allows
>>>>>>
>>>>> tests
>>>
>>>> to
>>>>>> fail such that you know it's an environment issue and not an actual
>>>>>>
>>>>> test
>>>
>>>> failure
>>>>>>
>>>>>> If you're not familiar with matchers, check out this quick tutorial
-
>>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>>
>>>>>> -Nathan
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <sean.xx.qiu@gmail.com>
>>>>>>
>>>>> wrote:
>>>
>>>>  Hi, all.
>>>>>>
>>>>>>  We had discussed the migration to testNG before and got some
>>>>>>>
>>>>>> conclusions
>>>
>>>> for
>>>>>>> grouping[1]
>>>>>>> including how to deal with boot path test[2]. Am i missing
>>>>>>>
>>>>>> something?
>>>
>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>>> I volunteer to carry out this job if no one objects.  Any other
>>>>>>> volunteers?
>>>>>>>
>>>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>>>> beginning.
>>>>>>> So our original junit tests can still work at the mean time when
>>>>>>> migrating.
>>>>>>> When one module's migration task is finished, we can judge the
>>>>>>>
>>>>>> result
>>>
>>>> to dertermine whether we should go on for other modules.
>>>>>>>
>>>>>>> Maybe we can create a branch for luni to start this work, shall
we?
>>>>>>> therefore there won't be any impact on other's development.
>>>>>>> Once it is completed in the branch, we could merge it back to
our
>>>>>>>
>>>>>> trunk.
>>>
>>>> Does it make sense?
>>>>>>>
>>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>> http://wiki.apache.org/harmony/Testing_Convention
>>>
>>>> [2]
>>>>>>>
>>>>>>>
>>>>>>>
>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>
>>>> [3]
>>>>>>>
>>>>>> http://testng.org/doc/documentation-main.html#annotations
>>>
>>>>  --
>>>>>>> Best Regards
>>>>>>> Sean, Xiao Xia Qiu
>>>>>>>  China Software Development Lab, IBM
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message