couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Test suite blocking release
Date Sat, 20 Mar 2010 18:17:33 GMT

On 20 Mar 2010, at 10:26, Noah Slater wrote:

> Jan, should this block the release? From what I can tell, it should.

I don't think a faulty test case should block the release.

Cheers
Jan
--

> 
> 
> 
> On 20 Mar 2010, at 11:32, Robert Dionne <dionne@dionne-associates.com> wrote:
> 
>> 
>> 
>> 
>> On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
>> 
>>> 
>>> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>>> 
>>>> 
>>>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>>>> 
>>>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <jan@apache.org>
wrote:
>>>>>> 
>>>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>>>>> 
>>>>>>>> We want to test the CouchDB code, not the browser's HTTP
handling.
>>>>>>> 
>>>>>>> Sure, but as one of CouchDB's primary interfaces is the browser,
it seems to makes sense that we would want to test how this works. Testing from the browser
allows us to test for and catch problems introduced by caching, etc - which is what our real
world users would be running into.
>>>>>>> 
>>>>>>> Unless I'm missing something?
>>>>>> 
>>>>>> I fully agree, but we should have a separate browser interaction
>>>>>> suite for that. The test suite is a very untypical browser client
and
>>>>>> doesn't really test real-world browser use-cases.
>>>>>> 
>>>>>> Cheers
>>>>>> Jan
>>>>>> --
>>>>> 
>>>>> +a bajillion.
>>>>> 
>>>> 
>>>> I prefer the browser tests because I'm much happier with JavaScript.
>>> 
>>> I'm not saying we should get rid of the browser tests. But intermittent errors
>>> in the current test suite are not to be worried about to block a release.
>> 
>> I agree with all the comments about separation of tests and so forth. This particular
changes test is not intermittent, it consistently fails (on my machine), enough that it's
a pleasant surprise when it succeeds. When running from the CLI I get the following:
>> 
>> not ok 10 changes expected '3', got '1'
>> 
>> When running in FF I also get the message above and occasionally:
>> 
>> • Exception raised: {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":"(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
>> 
>> I haven't looked into it closely to find the root cause, it might just be the test,
but it's definitely not intermittent. From the CLI it happens almost always
>> 
>> 
>> 
>>> 
>>> If we want proper browser client testing, we'd need an additional test suite
>>> that covers common and uncommon use-cases. I believe the current test
>>> suite is as untypical as a browser client can be.
>>> 
>>> Cheers
>>> Jan
>>> --
>>> 
>>> 
>>>> 
>>>> But maybe I'm crazy
>>>> 
>>>> 
>>>>> I think its important to maintain *some* tests in the browser to test
>>>>> its ability to use CouchDB as a client, but we should put more work
>>>>> into separating API tests and core tests.
>>>>> 
>>>>> Also, Zed Shaw has a very informative (and colorful) description of
>>>>> confounding factors [1]. Its about two thirds of the way down under a
>>>>> heading of "Confounding, Confounding, Confounding."
>>>>> 
>>>>> http://www.zedshaw.com/essays/programmer_stats.html
>>>> 
>>> 
>> 


Mime
View raw message