couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@me.com>
Subject Re: Test suite blocking release
Date Sat, 20 Mar 2010 16:52:03 GMT
Historically, test suite failures on Futon were release blockers. I'd  
like to get some consensus on if this is still the case. It's  
imperative we have a formal rule here, to help me, and to help in- 
between development. Once we've cleared that up, I suggest that we  
make this part of the documented release procedure. That way, asking  
for a release signifies that the test suites pass acceptably in the  
environments we have agreed the must pass in. I think this will reduce  
a significant amount of friction for future releases. If I get  
blocked, or stalled, in any way - that should indicate a process  
failure. Thoughts?



On 19 Mar 2010, at 23:25, Jan Lehnardt <jan@apache.org> wrote:

>
> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>
>>
>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>>
>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <jan@apache.org>  
>>> wrote:
>>>>
>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>>>
>>>>>
>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>>>
>>>>>> We want to test the CouchDB code, not the browser's HTTP  
>>>>>> handling.
>>>>>
>>>>> Sure, but as one of CouchDB's primary interfaces is the browser,  
>>>>> it seems to makes sense that we would want to test how this  
>>>>> works. Testing from the browser allows us to test for and catch  
>>>>> problems introduced by caching, etc - which is what our real  
>>>>> world users would be running into.
>>>>>
>>>>> Unless I'm missing something?
>>>>
>>>> I fully agree, but we should have a separate browser interaction
>>>> suite for that. The test suite is a very untypical browser client  
>>>> and
>>>> doesn't really test real-world browser use-cases.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>
>>> +a bajillion.
>>>
>>
>> I prefer the browser tests because I'm much happier with JavaScript.
>
> I'm not saying we should get rid of the browser tests. But  
> intermittent errors
> in the current test suite are not to be worried about to block a  
> release.
>
> If we want proper browser client testing, we'd need an additional  
> test suite
> that covers common and uncommon use-cases. I believe the current test
> suite is as untypical as a browser client can be.
>
> Cheers
> Jan
> --
>
>
>>
>> But maybe I'm crazy
>>
>>
>>> I think its important to maintain *some* tests in the browser to  
>>> test
>>> its ability to use CouchDB as a client, but we should put more work
>>> into separating API tests and core tests.
>>>
>>> Also, Zed Shaw has a very informative (and colorful) description of
>>> confounding factors [1]. Its about two thirds of the way down  
>>> under a
>>> heading of "Confounding, Confounding, Confounding."
>>>
>>> http://www.zedshaw.com/essays/programmer_stats.html
>>
>

Mime
View raw message