river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: Ignored tests
Date Tue, 24 Aug 2010 20:41:41 GMT
I'm not sure how much that would tell us, done on a bulk basis, because 
some of the tests will be specific to bugs that were found and fixed 
after then.

I will be doing something similar for individual tests, but taking into 
account what their comments tell me about which versions are expected to 
pass.

Patricia


On 8/24/2010 1:02 PM, Patrick Wright wrote:
> Hi Patricia
>
> Is there perhaps a solid baseline to test against, for example Jini
> 2.1 to see how many pass/fails we get?
>
> Thanks for all the hard work
> Patrick
>
> On Tue, Aug 24, 2010 at 9:58 PM, Patricia Shanahan<pats@acm.org>  wrote:
>> I ran a batch of the previously ignored QA tests overnight. I got 156 passes
>> and 64 failures. This is nowhere near as bad as it sounds, because many of
>> the failures were clusters of related tests failing in similar ways,
>> suggesting a single problem affecting the base infrastructure for the test
>> category. Some of the failures may relate to the known regression that Peter
>> is going to look at this week.
>>
>> Also, it is important to remember that the bugs may be in the tests, not in
>> the code under test. A test may be obsolete, depending on behavior that is
>> no longer supported.
>>
>> I do think there is a good enough chance that at least one of the failures
>> represents a real problem, and an opportunity to improve River, that I plan
>> to start a background activity looking at failed tests to see what is going
>> on. The objective is to do one of three things for each cluster of failures:
>>
>> 1. Fix River.
>>
>> 2. Fix the test.
>>
>> 3. Decide the test is unfixable, and delete it. There is no point spending
>> disk space, file transfer time, and test load time on tests we are never
>> going to run.
>>
>> Running the subset I did last night took about 15 hours, but that included a
>> lot of timeouts.
>>
>> Patricia
>>
>


Mime
View raw message