stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Black (JIRA)" <j...@apache.org>
Subject [jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
Date Fri, 21 Dec 2007 15:39:43 GMT

    [ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553961
] 

Andrew Black commented on STDCXX-683:
-------------------------------------

I think that a distinction between expected and unexpected NOUT states should be highlighted.
 This would permit us to easily spot non-regression tests which failed to produce any output.
 Without an XNOUT state, it would be necessary to check if each test is a regression test
before looking past an NOUT row.  In some cases, the regression notation (.stdcxx-NNN) may
be trimmed, due to the length of the test name, making it harder to determine if a test is
a regression test.  Perhaps it would make more sense to use XNOUT for unexpected NOUT results
and give it a yellow or red background.

An unrelated thought I'm having is that it might make sense to check how accessible the color
scheme is to someone who is red/green color blind.  One tool I tracked down that can be used
is http://colorfilter.wickline.org/ .  The observation I'm having is that the dark green background
used for the OK/XPASS states appears the same as the background used for the [SIG]<name>
state.

> implement notion of expected failures in the test suite
> -------------------------------------------------------
>
>                 Key: STDCXX-683
>                 URL: https://issues.apache.org/jira/browse/STDCXX-683
>             Project: C++ Standard Library
>          Issue Type: New Feature
>          Components: Test Driver, Tests
>    Affects Versions: 4.2.0
>            Reporter: Martin Sebor
>            Assignee: Martin Sebor
>            Priority: Critical
>         Attachments: codes.html, xcodes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to deal with
should be distinguished from failures that haven't been analyzed yet. For example, an example
program that fails to compile on an older target platform because of a compiler bug that we
can't find a simple/elegant workaround should be flagged as such in the test results. Similarly,
a test that fails one or more assertions due to compiler or libc bugs on a specific platform
(or a set of platforms) that we are unable to work around should be reported as such.
> This is important in order to reduce the currently fairly large number of unexpected
failures and to be able to make changes without having to worry about regressions as much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message