stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Sebor <>
Subject design of regression tests (was: Re: [jira] Updated: (STDCXX-436) [Linux] MB_LEN_MAX incorrect)
Date Fri, 14 Sep 2007 04:57:38 GMT
Travis, all,

I'm trying to decide how we should treat the latest regression test
for STDCXX-436:

Up until now our approach to "designing" regression tests has been
to simply copy the test case from each issue into the test suite
with only a minimum of changes. The test you've added goes far
beyond that, which makes it valuable because none of the other
macros is being tested, but at the same time it marks quite the
radical departure from the approach taken in all the other tests
which I hesitate to make without bringing it for discussion first.

On the one hand, when an issue comes in that points out a problem
with a feature so closely related to one or more others it makes
perfect sense to make sure that (and add tests for) the other
related features aren't broken, too. On the other hand, the name
of each regression test clearly indicates which bug it is designed
to exercise and when it should fail for some other reason (e.g.,
a regression in one of the other related macros) it would mislead
one into thinking that there's a problem with MB_LEN_MAX.

I suppose that my view on this is in cases like this, when the
bug report highlights a whole slew of features that aren't being
tested we should add an ordinary (unit) test for the whole area
and, if possible, also a regression test just for the bug. My
rationale for keeping the two separate (even at the cost of
duplicating some tested functionality) is that the bigger unit
test is more likely to be enhanced or tweaked in the future and
thus more likely to be subject to accidentally removing the test
for the bug (or otherwise "regressing"), while the regression
test is much more likely to be left  alone and consequently less
prone to such accidental regressions.

Opinions? Comments? Thoughts?


Travis Vitek (JIRA) wrote:
>      [
> Travis Vitek updated STDCXX-436:
> --------------------------------
>     Attachment: 18.limits.stdcxx-436.cpp
> Updated regression test compiles a simple program that simulates getconf and then uses
that program to get the values to compare against. The alternative was to just invoke getconv
as suggested, but that didn't work on windows, and getconf on some platforms doesn't provide
values for all constants [LONG_MIN and LONG_MAX on linux is one example].
>> [Linux] MB_LEN_MAX incorrect
>> ----------------------------
>>                 Key: STDCXX-436
>>                 URL:
>>             Project: C++ Standard Library
>>          Issue Type: Bug
>>          Components: 18. Language Support
>>    Affects Versions: 4.1.3
>>         Environment: gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)
>>            Reporter: Mark Brown
>>            Assignee: Travis Vitek
>>            Priority: Critical
>>             Fix For: 4.2
>>         Attachments: 18.limits.stdcxx-436.cpp, LIMITS.cpp.patch, stdcxx-436.patch
>> On my Linux system MB_LEN_MAX is normally defined to 16 but when I use the macro
in a program compiled with stdcxx the macro evaluates to 1. The test case goes like this:
>> $ cat test.cpp && make CPPOPTS="-DGETCONF_MB_LEN_MAX=`getconf MB_LEN_MAX`"
test && ./test
>> #include <assert.h>
>> #include <limits.h>
>> int main ()
>> {
>>     assert (MB_LEN_MAX == GETCONF_MB_LEN_MAX);
>> }
>> gcc -c -I/home/mbrown/stdcxx/include/ansi -D_RWSTDDEBUG    -I/home/mbrown/stdcxx/include
-I/home/mbrown/stdcxx-gcc-4.1.1-11s/include -I/home/mbrown/stdcxx/examples/include -DGETCONF_MB_LEN_MAX=16
-pedantic -nostdinc++ -g  -W -Wall -Wcast-qual -Winline -Wshadow -Wwrite-strings -Wno-long-long
-Wcast-align   test.cpp
>> gcc u.o -o u  -L/home/mbrown/stdcxx-gcc-4.1.1-11s/lib  -lstd11s -lsupc++ -lm 
>> test: test.cpp:6: int main(): Assertion `1 == 16' failed.
>> Aborted

View raw message