Mailing-List: contact stdcxx-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: stdcxx-dev@incubator.apache.org
Message-ID: <7314115.1198117003172.JavaMail.jira@brutus>
Date: Wed, 19 Dec 2007 18:16:43 -0800 (PST)
From: "Martin Sebor (JIRA)" <jira@apache.org>
To: stdcxx-dev@incubator.apache.org
Subject: [jira] Commented: (STDCXX-683) implement notion of expected
 failures in the test suite
In-Reply-To: <30925263.1198022396574.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553567 ] 

Martin Sebor commented on STDCXX-683:
-------------------------------------

Travis and I discussed this enhancement a few weeks back and here are the main goals we came up with:

1. It must be possible to mark up any failure in any component of the build and test process (i.e., any example, locale, or test) as an expected failure. This includes failure to compile, failure to link, exiting with a signal, exiting with a non-zero status, failure to produce the expected output (i.e., the DIFF status for examples), and individual failed assertions (in tests).

2. For each such markup/failure, it must be possible to differentiate between the same set of platforms and configurations of the library that are distinguished by the current build and test reporting system. That a mechanism must be provided to make it possible to specify that a particular failure (for example, a failure to compile) is expected to occur only on a specific platform (e.g., with IBM XLC++ 7.0.0.9 on AIX 5.3) or a set of platforms (e.g., with all versions of gcc prior to 3.2.4, or with any compiler on Solaris), and only in a particular configuration of the library (such as 12D, or optimized, shared, thread-safe, wide or 64-bit) or a set of such configurations (e.g., all 64-bit ones).

3. For tests, it must be possible to mark up individual failed assertions (and any other types of diagnostics) as expected using their sequential numbers (assigned to them by the test driver) at runtime.

> implement notion of expected failures in the test suite
> -------------------------------------------------------
>
>                 Key: STDCXX-683
>                 URL: https://issues.apache.org/jira/browse/STDCXX-683
>             Project: C++ Standard Library
>          Issue Type: New Feature
>          Components: Test Driver, Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Martin Sebor
>            Priority: Critical
>         Attachments: codes.html, xcodes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to deal with should be distinguished from failures that haven't been analyzed yet. For example, an example program that fails to compile on an older target platform because of a compiler bug that we can't find a simple/elegant workaround should be flagged as such in the test results. Similarly, a test that fails one or more assertions due to compiler or libc bugs on a specific platform (or a set of platforms) that we are unable to work around should be reported as such.
> This is important in order to reduce the currently fairly large number of unexpected failures and to be able to make changes without having to worry about regressions as much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.