couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dionne <dio...@dionne-associates.com>
Subject Re: Test suite blocking release
Date Sat, 20 Mar 2010 11:32:38 GMT



On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:

> 
> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
> 
>> 
>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>> 
>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <jan@apache.org> wrote:
>>>> 
>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>>> 
>>>>> 
>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>>> 
>>>>>> We want to test the CouchDB code, not the browser's HTTP handling.
>>>>> 
>>>>> Sure, but as one of CouchDB's primary interfaces is the browser, it seems
to makes sense that we would want to test how this works. Testing from the browser allows
us to test for and catch problems introduced by caching, etc - which is what our real world
users would be running into.
>>>>> 
>>>>> Unless I'm missing something?
>>>> 
>>>> I fully agree, but we should have a separate browser interaction
>>>> suite for that. The test suite is a very untypical browser client and
>>>> doesn't really test real-world browser use-cases.
>>>> 
>>>> Cheers
>>>> Jan
>>>> --
>>> 
>>> +a bajillion.
>>> 
>> 
>> I prefer the browser tests because I'm much happier with JavaScript.
> 
> I'm not saying we should get rid of the browser tests. But intermittent errors
> in the current test suite are not to be worried about to block a release.

I agree with all the comments about separation of tests and so forth. This particular changes
test is not intermittent, it consistently fails (on my machine), enough that it's a pleasant
surprise when it succeeds. When running from the CLI I get the following:

not ok 10 changes expected '3', got '1'

When running in FF I also get the message above and occasionally:

• Exception raised: {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":"(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}

I haven't looked into it closely to find the root cause, it might just be the test, but it's
definitely not intermittent. From the CLI it happens almost always



> 
> If we want proper browser client testing, we'd need an additional test suite
> that covers common and uncommon use-cases. I believe the current test
> suite is as untypical as a browser client can be.
> 
> Cheers
> Jan
> --
> 
> 
>> 
>> But maybe I'm crazy
>> 
>> 
>>> I think its important to maintain *some* tests in the browser to test
>>> its ability to use CouchDB as a client, but we should put more work
>>> into separating API tests and core tests.
>>> 
>>> Also, Zed Shaw has a very informative (and colorful) description of
>>> confounding factors [1]. Its about two thirds of the way down under a
>>> heading of "Confounding, Confounding, Confounding."
>>> 
>>> http://www.zedshaw.com/essays/programmer_stats.html
>> 
> 


Mime
View raw message