river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricia Shanahan <p...@acm.org>
Subject Re: Ignored tests
Date Wed, 25 Aug 2010 21:53:53 GMT
Can you give any guidance on how to find out which tests need what 
infrastructure? Is it documented somewhere? I'm still learning my way 
around the River files.

Also, I'm interested in tests that fail unexpectedly, especially any 
tests that have regressed or fail intermittently without related source 
code changes.

I have a suspicion, based on source code reading, of a race condition in 
ServiceDiscoveryManager, and problems related to retries in some 
subclasses of RetryTask. If these problems are real they would tend to 
lead to unreproducible, intermittent failures rather than solid failures.

Patricia


On 8/25/2010 2:30 PM, Jonathan Costers wrote:
> There is one more test category that we could add to the list that is used
> by Hudson: "renewalmanager".
> All the other categories have one or more issues (I have run all these tests
> myself many, many times), mostly because of missing infrastructure, but some
> also fail unexpectedly.
>
>
> 2010/8/24 Patricia Shanahan<pats@acm.org>
>
>> I'm not sure how much that would tell us, done on a bulk basis, because
>> some of the tests will be specific to bugs that were found and fixed after
>> then.
>>
>> I will be doing something similar for individual tests, but taking into
>> account what their comments tell me about which versions are expected to
>> pass.
>>
>> Patricia
>>
>>
>>
>> On 8/24/2010 1:02 PM, Patrick Wright wrote:
>>
>>> Hi Patricia
>>>
>>> Is there perhaps a solid baseline to test against, for example Jini
>>> 2.1 to see how many pass/fails we get?
>>>
>>> Thanks for all the hard work
>>> Patrick
>>>
>>> On Tue, Aug 24, 2010 at 9:58 PM, Patricia Shanahan<pats@acm.org>   wrote:
>>>
>>>> I ran a batch of the previously ignored QA tests overnight. I got 156
>>>> passes
>>>> and 64 failures. This is nowhere near as bad as it sounds, because many
>>>> of
>>>> the failures were clusters of related tests failing in similar ways,
>>>> suggesting a single problem affecting the base infrastructure for the
>>>> test
>>>> category. Some of the failures may relate to the known regression that
>>>> Peter
>>>> is going to look at this week.
>>>>
>>>> Also, it is important to remember that the bugs may be in the tests, not
>>>> in
>>>> the code under test. A test may be obsolete, depending on behavior that
>>>> is
>>>> no longer supported.
>>>>
>>>> I do think there is a good enough chance that at least one of the
>>>> failures
>>>> represents a real problem, and an opportunity to improve River, that I
>>>> plan
>>>> to start a background activity looking at failed tests to see what is
>>>> going
>>>> on. The objective is to do one of three things for each cluster of
>>>> failures:
>>>>
>>>> 1. Fix River.
>>>>
>>>> 2. Fix the test.
>>>>
>>>> 3. Decide the test is unfixable, and delete it. There is no point
>>>> spending
>>>> disk space, file transfer time, and test load time on tests we are never
>>>> going to run.
>>>>
>>>> Running the subset I did last night took about 15 hours, but that
>>>> included a
>>>> lot of timeouts.
>>>>
>>>> Patricia
>>>>
>>>>
>>>
>>
>


Mime
View raw message