incubator-stdcxx-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lemings <eric.lemi...@roguewave.com>
Subject Re: Regression tests file structure
Date Sun, 27 Apr 2008 00:51:40 GMT

On Apr 26, 2008, at 2:42 PM, Martin Sebor wrote:
> Eric Lemings wrote:
>>
>>> -----Original Message-----
>>> From: Martin Sebor [mailto:sebor@roguewave.com] Sent: Friday,  
>>> April 25, 2008 1:03 PM
>>> To: dev@stdcxx.apache.org
>>> Subject: Re: Regression tests file structure
>>>
>>> Eric Lemings wrote:
>>>> Not that it really matters but I noticed lots of rather
>>> small programs
>>>> for the regression tests.  Couldn't these be consolidated into  
>>>> fewer
>>>> source files, by component perhaps, with a function for
>>> each individual
>>>> regression test?  Just a thought.
>>> I'm sure they could, and it would probably benefit our build
>>> times. The problem is that a severe failure in one of them
>>> (build or runtime) would cause the rest of them to fail too.
>>> Getting around it would require non-trivial changes to our
>>> build harness (e.g., build each test into a separate object
>>> file, linking them all together into a single executable,
>>> and on a runtime failure in one re-link all those that
>>> didn't get to run into a new executable, and repeat).
>> Mmm, that's not quite what I had in mind.  Here's an example of  
>> what I
>> had in mind:
>> test/regress/21.string.cpp:
>> ...
>> static void
>> test_stdcxx_162 () {
>> ...
>> }
>> static void
>> test_stdcxx_231 () {
>> ...
>> }
>> static void
>> test_stdcxx_466 () {
>> ...
>> }
>> static void
>> run_tests () {
>> test_stdcxx_162 ();
>> test_stdcxx_231 ();
>> test_stdcxx_466 ();
>> }
>> All regression tests related to the same component (i.e. clause) are
>> contained in the same source file.  That wouldn't require build/test
>> harness changes, would it?  And yes a build or run failure would  
>> cause
>> them all to fail but I'd see that as real incentive for correcting  
>> the
>> failure.  :)
>
> We certainly could structure the tests this way. It would alleviate
> the fatal failure problem to some extent but it wouldn't completely
> eliminate it.
>
> I'm not sure that the incentive argument is any more powerful for
> regression tests than it is for the others, and look at how many
> failures we still have in the test suite. It would need to be
> weighed against the benefits of this restructuring. What do you
> see as the main benefits of this change?

Better organization mostly.  Fewer test programs with only one test  
case.  I figure if you're adding a new test program, it should have at  
least a few test cases, not just one.  If you're just adding just a  
new test case, there's probably a related test program already in the  
test suite.

No, these aren't huge benefits but neither are the costs.  As I said  
opening the thread, "not that it really matters..."

Brad.


Mime
View raw message