stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Sebor (JIRA)" <>
Subject [jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
Date Thu, 20 Dec 2007 02:16:43 GMT


Martin Sebor commented on STDCXX-683:

Travis and I discussed this enhancement a few weeks back and here are the main goals we came
up with:

1. It must be possible to mark up any failure in any component of the build and test process
(i.e., any example, locale, or test) as an expected failure. This includes failure to compile,
failure to link, exiting with a signal, exiting with a non-zero status, failure to produce
the expected output (i.e., the DIFF status for examples), and individual failed assertions
(in tests).

2. For each such markup/failure, it must be possible to differentiate between the same set
of platforms and configurations of the library that are distinguished by the current build
and test reporting system. That a mechanism must be provided to make it possible to specify
that a particular failure (for example, a failure to compile) is expected to occur only on
a specific platform (e.g., with IBM XLC++ on AIX 5.3) or a set of platforms (e.g.,
with all versions of gcc prior to 3.2.4, or with any compiler on Solaris), and only in a particular
configuration of the library (such as 12D, or optimized, shared, thread-safe, wide or 64-bit)
or a set of such configurations (e.g., all 64-bit ones).

3. For tests, it must be possible to mark up individual failed assertions (and any other types
of diagnostics) as expected using their sequential numbers (assigned to them by the test driver)
at runtime.

> implement notion of expected failures in the test suite
> -------------------------------------------------------
>                 Key: STDCXX-683
>                 URL:
>             Project: C++ Standard Library
>          Issue Type: New Feature
>          Components: Test Driver, Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Martin Sebor
>            Priority: Critical
>         Attachments: codes.html, xcodes.html
> Tests (or examples) that fail for known reasons that we haven't been able to deal with
should be distinguished from failures that haven't been analyzed yet. For example, an example
program that fails to compile on an older target platform because of a compiler bug that we
can't find a simple/elegant workaround should be flagged as such in the test results. Similarly,
a test that fails one or more assertions due to compiler or libc bugs on a specific platform
(or a set of platforms) that we are unable to work around should be reported as such.
> This is important in order to reduce the currently fairly large number of unexpected
failures and to be able to make changes without having to worry about regressions as much.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message