subversion-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Schaber <m.scha...@codesys.com>
Subject Proposal for separating Tests into groups
Date Thu, 13 Jun 2013 13:20:11 GMT
Hi,

This are two alternative proposals for the test suite:

Rationale: Developers restrain from implementing some tests because they just take to much
time to run at every commit.

Other test systems like JUnit, NUnit or the CODESYS Test Manager come with ways to select
unit tests by category, so we could implement something similar with our tests.

1) Just two test sets: simple and extended tests:
Tests which take a lot of time and cover areas which are unlikely to break can be marked as
extended tests. For Python, there will be an @extended decorator, for the C tests we'll have
macros like SVN_TEST_EXTENDED_PASS.

Then running the test with an command line option --simple will skip those tests. Explicitly
mentioning extended tests by number will still run them, and can be combined with the --simple
flag.

Continous integration systems and the tests before signing an release should still execute
the full test suite.

But before a small, not destabilizing commit (use common sense here), only running the non-extended
tests is mandatory (and maybe the extended tests covering that specific area.)

For make check, it would be an SIMPLE=true variable.

Alternatively:
2) Test Categories:
A set of categories is defined in a central place. 

Examples for such categories could be:
- Smoke: For smoke tests, only the most important.
- Fsfs: Tests covering only the FSFS specific code.
- Offline: Tests which do not contact the server.
- Repository: Tests which cover the repository, without a client involved (e. G. svnadmin)

Each test then gets attributed with the categories which are valid for this test.

When running the tests, one could pass a parameter to run only the tests which are attributed
with at least one of the given flags. For example, if you changed something in FSFS, "--categories=Smoke,Fsfs"
would run the smoke tests and the FSFS tests. A second "--exclude=Repository,5,7" switch could
be used to exclude test categories as well as single tests by number.

For make check, we'd have a CATEGORIES and EXCLUDE variables.



Both ideas are still in a very raw, unpolished state, and may be discussed today live on the
meeting.




Best regards

Markus Schaber

(This email was sent from a mobile device...)

CODESYS® a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions
________________________________
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber@codesys.com | Web: codesys.com
CODESYS internet forum: forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten
HRB 6186 | Tax ID No.: DE 167014915

Mime
View raw message