couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@me.com>
Subject Re: Test suite blocking release
Date Sat, 20 Mar 2010 15:26:32 GMT
Jan, should this block the release? From what I can tell, it should.



On 20 Mar 2010, at 11:32, Robert Dionne <dionne@dionne-associates.com>  
wrote:

>
>
>
> On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
>
>>
>> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>>
>>>
>>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>>>
>>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <jan@apache.org>  
>>>> wrote:
>>>>>
>>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>>>>
>>>>>>
>>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>>>>
>>>>>>> We want to test the CouchDB code, not the browser's HTTP  
>>>>>>> handling.
>>>>>>
>>>>>> Sure, but as one of CouchDB's primary interfaces is the  
>>>>>> browser, it seems to makes sense that we would want to test how 

>>>>>> this works. Testing from the browser allows us to test for and  
>>>>>> catch problems introduced by caching, etc - which is what our  
>>>>>> real world users would be running into.
>>>>>>
>>>>>> Unless I'm missing something?
>>>>>
>>>>> I fully agree, but we should have a separate browser interaction
>>>>> suite for that. The test suite is a very untypical browser  
>>>>> client and
>>>>> doesn't really test real-world browser use-cases.
>>>>>
>>>>> Cheers
>>>>> Jan
>>>>> --
>>>>
>>>> +a bajillion.
>>>>
>>>
>>> I prefer the browser tests because I'm much happier with JavaScript.
>>
>> I'm not saying we should get rid of the browser tests. But  
>> intermittent errors
>> in the current test suite are not to be worried about to block a  
>> release.
>
> I agree with all the comments about separation of tests and so  
> forth. This particular changes test is not intermittent, it  
> consistently fails (on my machine), enough that it's a pleasant  
> surprise when it succeeds. When running from the CLI I get the  
> following:
>
> not ok 10 changes expected '3', got '1'
>
> When running in FF I also get the message above and occasionally:
>
> • Exception raised: {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11

> .0","lineNumber":154,"stack":"(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a

> "}
>
> I haven't looked into it closely to find the root cause, it might  
> just be the test, but it's definitely not intermittent. From the CLI  
> it happens almost always
>
>
>
>>
>> If we want proper browser client testing, we'd need an additional  
>> test suite
>> that covers common and uncommon use-cases. I believe the current test
>> suite is as untypical as a browser client can be.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>>
>>> But maybe I'm crazy
>>>
>>>
>>>> I think its important to maintain *some* tests in the browser to  
>>>> test
>>>> its ability to use CouchDB as a client, but we should put more work
>>>> into separating API tests and core tests.
>>>>
>>>> Also, Zed Shaw has a very informative (and colorful) description of
>>>> confounding factors [1]. Its about two thirds of the way down  
>>>> under a
>>>> heading of "Confounding, Confounding, Confounding."
>>>>
>>>> http://www.zedshaw.com/essays/programmer_stats.html
>>>
>>
>

Mime
View raw message