stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Sebor <se...@roguewave.com>
Subject Re: Regression tests file structure
Date Sat, 26 Apr 2008 20:42:13 GMT
Eric Lemings wrote:
>  
> 
>> -----Original Message-----
>> From: Martin Sebor [mailto:sebor@roguewave.com] 
>> Sent: Friday, April 25, 2008 1:03 PM
>> To: dev@stdcxx.apache.org
>> Subject: Re: Regression tests file structure
>>
>> Eric Lemings wrote:
>>>  
>>> Not that it really matters but I noticed lots of rather 
>> small programs
>>> for the regression tests.  Couldn't these be consolidated into fewer
>>> source files, by component perhaps, with a function for 
>> each individual
>>> regression test?  Just a thought.
>> I'm sure they could, and it would probably benefit our build
>> times. The problem is that a severe failure in one of them
>> (build or runtime) would cause the rest of them to fail too.
>> Getting around it would require non-trivial changes to our
>> build harness (e.g., build each test into a separate object
>> file, linking them all together into a single executable,
>> and on a runtime failure in one re-link all those that
>> didn't get to run into a new executable, and repeat).
> 
> Mmm, that's not quite what I had in mind.  Here's an example of what I
> had in mind:
> 
> test/regress/21.string.cpp:
> ...
> static void
> test_stdcxx_162 () {
> ...
> }
> 
> static void
> test_stdcxx_231 () {
> ...
> }
> 
> static void
> test_stdcxx_466 () {
> ...
> }
> 
> static void
> run_tests () {
>   test_stdcxx_162 ();
>   test_stdcxx_231 ();
>   test_stdcxx_466 ();
> }
> 
> All regression tests related to the same component (i.e. clause) are
> contained in the same source file.  That wouldn't require build/test
> harness changes, would it?  And yes a build or run failure would cause
> them all to fail but I'd see that as real incentive for correcting the
> failure.  :)

We certainly could structure the tests this way. It would alleviate
the fatal failure problem to some extent but it wouldn't completely
eliminate it.

I'm not sure that the incentive argument is any more powerful for
regression tests than it is for the others, and look at how many
failures we still have in the test suite. It would need to be
weighed against the benefits of this restructuring. What do you
see as the main benefits of this change?

Martin


Mime
View raw message