couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@me.com>
Subject Re: Test suite blocking release
Date Mon, 22 Mar 2010 15:15:12 GMT
Yes, but it's very noisy, and I wanted to get absolute clarification. Thanks. I'll do it now.

On 22 Mar 2010, at 15:08, Jan Lehnardt wrote:

> 
> On 22 Mar 2010, at 04:12, Noah Slater wrote:
> 
>> So, are we officially good to go?
>> 
>> Can I upload the artefact from the current tag, or do I need to retag the 0.11.x
branch?
> 
> You'll need to re-tag. Are you not on commits@? :)
> 
> Cheers
> Jan
> --
> 
> 
> 
>> 
>> Thanks.
>> 
>> On 21 Mar 2010, at 19:00, Jan Lehnardt wrote:
>> 
>>> 
>>> On 21 Mar 2010, at 13:38, Robert Dionne wrote:
>>> 
>>>> Ok Noah,  This is only a test case issue, and not in the changes code as
I though. Jan found the issue and it works fine for me now in both FF and CLI. -- Bob
>>> 
>>> As an added bonus, the test suite should now work 100% in WebKit (for me at least
:)
>>> 
>>> Cheers
>>> Jan
>>> --
>>> 
>>> 
>>>> 
>>>> 
>>>> On Mar 21, 2010, at 1:30 PM, Jan Lehnardt wrote:
>>>> 
>>>>> 
>>>>> On 21 Mar 2010, at 12:24, Robert Dionne wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>>>>>>> 
>>>>>>>> What are the CLI tests, if not the etap tests? Are they integrated
into the build system?
>>>>>>> 
>>>>>>> The CLI tests are the same as the browser tests, just run through
our couchjs binary
>>>>>>> that has custom HTTP extensions to make the xhr work. At this
point I don't think it
>>>>>>> is reliable enough to mimic browser behaviour and that we shouldn't
use it for vetting
>>>>>>> the state of the code.
>>>>>> 
>>>>>> This is likely true, but in this particular case I think there's
a bug in the changes code (that I'm trying to dig out). It's nice that it works on your machine
but on my machine, using FF, it fails often enough. Moreover it's been around for a long time
now so I figure it's worth figuring out. 
>>>>>> 
>>>>>> I don't have a dog in this fight (.ie. a paying customer) so this
failure doesn't bother me. With respect to policy, given the various bogocities of browsers,
I'd recommend something like these CLI tests plus the etaps ought to be the "official"  tests
for vetting, and part of the build
>>>>> 
>>>>> Not that I disagree, but part (most?) of the appeal of the browser based
tests are that they run in a real-world client instead of the lab that is couchjs+http :)
>>>>> 
>>>>> Cheers
>>>>> Jan
>>>>> --
>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> It is very useful when developing new code to not have to switch
to and reload the
>>>>>>> browser over and over again.
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Jan
>>>>>>> --
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>>>>>>>> 
>>>>>>>>>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater
<nslater@me.com> wrote:
>>>>>>>>>>>>> I think faulty test case should block
the release, if I am to have any
>>>>>>>>>>>>> future sanity preparing releases. I don't
want to delay and longer, so if
>>>>>>>>>>>>> you guys are absolutely sure this is
a test error and not code error, then I
>>>>>>>>>>>>> propose that the test be commented out.
Our tests form a contract between
>>>>>>>>>>>>> us, internally, and our users. If that
contract has a bug, it should be
>>>>>>>>>>>>> removed or fixed - or it simply dilutes
the importance of contract. If some
>>>>>>>>>>>>> one comments out the test, and we agree
it is not indicative of an important
>>>>>>>>>>>>> bug, I can call the vote within hours.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I'd have to agree on this. From the point
of view of a release, if a
>>>>>>>>>>>> test reports a failure then it should be
made to not report a failure.
>>>>>>>>>>>> If that's accomplished by disabling it, then
there will be a commit
>>>>>>>>>>>> with a message that explains why it was disabled
and etc and such on
>>>>>>>>>>>> and so forth.
>>>>>>>>>>> 
>>>>>>>>>>> I'd do that if the test was failing for me :)
>>>>>>>>>> 
>>>>>>>>>> it's not failing for you when you run changes.js
with the CLI ?  Fails for me every time. 
>>>>>>>>> 
>>>>>>>>> I don't consider the CLI tests as part of the official
test suite just yet.
>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> Jan
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Anyway I poked at this a bit yesterday and am not
100% sure the issue is in the test. I tried putting a sleep in with no luck. If my understanding
of the JS is correct, CouchDB is supposed to be synchronous so it's not timing.
>>>>>>>>>> 
>>>>>>>>>> If someone could comment on the test itself it would
be helpful. The section of the code that fails:
>>>>>>>>>> 
>>>>>>>>>> // changes get all_docs style with deleted docs
>>>>>>>>>> var doc = {a:1};
>>>>>>>>>> db.save(doc);
>>>>>>>>>> db.deleteDoc(doc);
>>>>>>>>>> var req = CouchDB.request("GET", 
>>>>>>>>>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs");
>>>>>>>>>> var resp = JSON.parse(req.responseText);
>>>>>>>>>> TEquals(3, resp.results.length, "should return matching
rows");
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> seems odd to me. all_docs as I read the code will
return docs with deletes and conflicts but in this call the filter bop will not apply to the
doc {a:1} so I'm not sure what this delete prior to the call is about. Anyway I can make it
fail in the debugger so perhaps I can find the root cause.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Cheers
>>>>>>>>>>> Jan
>>>>>>>>>>> --
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


Mime
View raw message