harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Liang <richard.lian...@gmail.com>
Subject Re: [classlib][TestNG] groups of Harmony test
Date Wed, 30 Aug 2006 02:35:49 GMT


Richard Liang wrote:
>
>
> Tony Wu wrote:
>> I think we can treat something like *os.any*,*type.api* as default
>> groups.Itwill make our logic more clear and bring convenience to
>> testcase writing :)
>> for example, an os.<platformid> group indicates that it is designed for
>> <paltformid> only. If there is no group like os.<platformid> here, 
>> this is a
>> *os.any* test.
>>
>> so if we want to run all *api* tests on *win32* platform which is *not
>> broken*, we could write the testng.xml like this:
>> <groups>
>>           <run>
>>               <include name=".*" />
>>               <exclude name="type.impl" />
>>               <exclude name="os\.(?!win\.IA32).*" />
>>               <exclude name="state.broken" />
>>               <exclude name="state.broken.win.IA32" />
>>           </run>
>>       </groups>
>>
> Hello Tony,
>
> It's a good idea. However, we shall define a "default" group which 
> means "os.any, type.impl, level.unit". Thanks a lot.
>
And even the "default" group is not necessary by removing the "<include 
name=".*" /> "

      <groups>
          <run>
              <exclude name="type.impl" />
              <exclude name="os\.(?!win\.IA32).*" />
              <exclude name="state.broken" />
              <exclude name="state.broken.win.IA32" />
          </run>
      </groups>


Richard
> Best regards,
> Richard
>
>>
>> 2006/8/29, Vladimir Ivanov <ivavladimir@gmail.com>:
>>>
>>> Also some tag for regression tests should be added.
>>> thanks, Vladimir
>>>
>>>
>>> On 8/28/06, Richard Liang <richard.liangyx@gmail.com> wrote:
>>> >
>>> >
>>> >
>>> > Richard Liang wrote:
>>> > > Hello All,
>>> > >
>>> > > Now let's talk about the TestNG groups. I have read the related
>>> > > threads which posted by George, Vladimir Ivanov and Alexei 
>>> Zakharov.
>>> > > All of them are good discussion about TestNG groups.
>>> > >
>>> > > IMHO, we may define Harmony test groups according the following 4
>>> > > dimensions:
>>> > >
>>> > > 1) [Platform] os.any, os.<platform id>
>>> > > *os.any* - group of tests which pass on any platform. IMHO, most of
>>> > > our tests should be in this group.
>>> > > *os.<platform id>* - group of tests which are designed for one
>>> > > specific platform. A test may be in more than one of the groups. 
>>> e.g.,
>>> > > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>> > >
>>> > >    ** os.any and os.<platform id> are mutually exclusive, that
is,
>>> > > tests in os.any group should not be in os.win.IA32.
>>> > >
>>> > > 2) [Test state] state.broken, state.broken.<platform id>
>>> > > *state.broken* - group of tests which fail on every platform, 
>>> because
>>> > > of bugs of tests or implementation. We need to fix the bugs of 
>>> tests
>>> > > or implementation to make them pass.
>>> > > *state.broken.<platform id>* - groups of test which only fail
on 
>>> one
>>> > > specific platform. A test may be in more than one of the groups. 
>>> e.g.,
>>> > > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>> > >
>>> > >     **state.broken.<platform id> group may be used as a 
>>> convenient way
>>> > > to indicate that a test is platform-specific. e.g., If we 
>>> support 10
>>> > > platforms, and one test are designed for 9 platforms except for 
>>> MacOS,
>>> > > instead of list 9 os.<platform id>, we can just use 
>>> state.broken.MacOS
>>> > >
>>> > > 3) [Test type] type.api , type.impl
>>> > > *type.api* - group of tests which are tests for APIs in the Java
>>> > > Specification
>>> > > *type.impl* - groups of tests which are tests for Harmony-specific
>>> > > implementation
>>> > >
>>> > >     ** type.api and type.impl are also mutually exclusive.
>>> > >
>>> > > 4) [Test Level] level.unit, level.integration, level.system,
>>> > > level.stress, etc. (Levels of Test refer to the increase in 
>>> complexity
>>> > > as moving through test cycle. )
>>> > >    ** A test may be in more than one of the groups.
>>> > >    ** In fact, some tests such as System tests are the 
>>> verification of
>>> > > the entire system.  Maybe we'll put them into a separate project.
>>> > > e.g., harmony/enhanced/SVT (System Verification Test).
>>> > >
>>> > > If we want to run all the unit test for APIs on windows, we may use
>>> > > TestNG groups to select the tests:
>>> > >        <groups>
>>> > >            <run>
>>> > >                <include name="os.any" />
>>> > >                <include name="type.api" />
>>> > >                <include name="os.win.IA32" />
>>> > >                <exclude name=" state.broken" />
>>> > >                <exclude name="state.broken.win.IA32" />
>>> > >            </run>
>>> > >        </groups>
>>> > >
>>> > Hello All,
>>> >
>>> > I'm sorry. It seems that the example does not work. I will try to 
>>> figure
>>> > another example soon. ;-)
>>> >
>>> > Best regards,
>>> > Richard
>>> > >
>>> > > Well, I think our most of existing tests are in the groups of
>>> > > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to 
>>> add a
>>> > > new option "-groups" for its JUnitConverter which allow us to 
>>> specify
>>> > > the test groups when migrate from JUnit test to TestNG test.
>>> > >
>>> > > Thanks for reading so far, and I will highly appreciate your 
>>> comments
>>> > > or suggestion.  ;-)
>>> > >
>>> >
>>> > --
>>> > Richard Liang
>>> > China Software Development Lab, IBM
>>> >
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> > For additional commands, e-mail: 
>>> harmony-dev-help@incubator.apache.org
>>> >
>>> >
>>>
>>>
>>
>>
>

-- 
Richard Liang
China Software Development Lab, IBM 



---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message