Mailing-List: contact dev-help@stdcxx.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@stdcxx.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <E287871D-3A0C-4B1F-91E6-A4CEF4EE5291@roguewave.com>
From: Eric Lemings <eric.lemings@roguewave.com>
To: dev@stdcxx.apache.org
In-Reply-To: <481393A5.40703@roguewave.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v919.2)
Subject: Re: Regression tests file structure
Date: Sat, 26 Apr 2008 18:51:40 -0600
References: 
 <CFFDD219128FD94FB4F92B99F52D0A49010A3730@exchmail01.Blue.Roguewave.Com>
 <48122AE1.1060805@roguewave.com>
 <CFFDD219128FD94FB4F92B99F52D0A49010A3748@exchmail01.Blue.Roguewave.Com>
 <481393A5.40703@roguewave.com>


On Apr 26, 2008, at 2:42 PM, Martin Sebor wrote:
> Eric Lemings wrote:
>>
>>> -----Original Message-----
>>> From: Martin Sebor [mailto:sebor@roguewave.com] Sent: Friday,  
>>> April 25, 2008 1:03 PM
>>> To: dev@stdcxx.apache.org
>>> Subject: Re: Regression tests file structure
>>>
>>> Eric Lemings wrote:
>>>> Not that it really matters but I noticed lots of rather
>>> small programs
>>>> for the regression tests.  Couldn't these be consolidated into  
>>>> fewer
>>>> source files, by component perhaps, with a function for
>>> each individual
>>>> regression test?  Just a thought.
>>> I'm sure they could, and it would probably benefit our build
>>> times. The problem is that a severe failure in one of them
>>> (build or runtime) would cause the rest of them to fail too.
>>> Getting around it would require non-trivial changes to our
>>> build harness (e.g., build each test into a separate object
>>> file, linking them all together into a single executable,
>>> and on a runtime failure in one re-link all those that
>>> didn't get to run into a new executable, and repeat).
>> Mmm, that's not quite what I had in mind.  Here's an example of  
>> what I
>> had in mind:
>> test/regress/21.string.cpp:
>> ...
>> static void
>> test_stdcxx_162 () {
>> ...
>> }
>> static void
>> test_stdcxx_231 () {
>> ...
>> }
>> static void
>> test_stdcxx_466 () {
>> ...
>> }
>> static void
>> run_tests () {
>> test_stdcxx_162 ();
>> test_stdcxx_231 ();
>> test_stdcxx_466 ();
>> }
>> All regression tests related to the same component (i.e. clause) are
>> contained in the same source file.  That wouldn't require build/test
>> harness changes, would it?  And yes a build or run failure would  
>> cause
>> them all to fail but I'd see that as real incentive for correcting  
>> the
>> failure.  :)
>
> We certainly could structure the tests this way. It would alleviate
> the fatal failure problem to some extent but it wouldn't completely
> eliminate it.
>
> I'm not sure that the incentive argument is any more powerful for
> regression tests than it is for the others, and look at how many
> failures we still have in the test suite. It would need to be
> weighed against the benefits of this restructuring. What do you
> see as the main benefits of this change?

Better organization mostly.  Fewer test programs with only one test  
case.  I figure if you're adding a new test program, it should have at  
least a few test cases, not just one.  If you're just adding just a  
new test case, there's probably a related test program already in the  
test suite.

No, these aren't huge benefits but neither are the costs.  As I said  
opening the thread, "not that it really matters..."

Brad.