Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: Test suite blocking release
From: Jan Lehnardt <jan@apache.org>
In-Reply-To: <30A3B728-5F64-4C19-BA04-3E75AFEE3778@dionne-associates.com>
Date: Sun, 21 Mar 2010 12:30:20 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <2D873643-D038-494E-9C67-863E4241F9CC@apache.org>
References: <2733FAEB-FB2C-4CFF-A9B6-A60850F447ED@me.com>
 <700C5F25-8235-4EFA-A3AE-5C330F5FE394@apache.org>
 <e2111bbb1003191143x70b4cc59q2f032b8e15e3c905@mail.gmail.com>
 <24496B6A-5808-486C-9C95-F0B9E7C55D60@gmail.com>
 <6AE37542-8E22-4644-95F6-6527B08D4736@apache.org>
 <A13BCD0F-3B1D-41B9-8E73-8359153F3D15@dionne-associates.com>
 <66918683-1D1D-4C00-8A31-E11684CEA73D@me.com>
 <2A53C38E-C053-44D1-BFA3-ECC4C049B99F@apache.org>
 <b7cd8ed11003201119j534097bi3d592918ae8bd40a@mail.gmail.com>
 <4D84D3B7-F601-4C17-94CF-D1053E3A4D8A@me.com>
 <e2111bbb1003201806m2d6ceba4xe89da663c032bf9b@mail.gmail.com>
 <ED6A5E13-4408-4FEE-9521-EC7C7A5F6B75@apache.org>
 <DEFF43B6-5365-472A-B184-C8FC0B38107D@dionne-associates.com>
 <F1346030-E4A2-4C6A-B27B-9BEE5FDF8953@apache.org>
 <4D831DF2-68D2-4BF1-A2A8-77FC3BB0AD05@me.com>
 <AB014624-0879-4355-920C-C9A19499BCC3@apache.org>
 <30A3B728-5F64-4C19-BA04-3E75AFEE3778@dionne-associates.com>
To: dev@couchdb.apache.org


On 21 Mar 2010, at 12:24, Robert Dionne wrote:

>=20
>=20
>=20
>=20
> On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote:
>=20
>>=20
>> On 21 Mar 2010, at 12:10, Noah Slater wrote:
>>=20
>>> What are the CLI tests, if not the etap tests? Are they integrated =
into the build system?
>>=20
>> The CLI tests are the same as the browser tests, just run through our =
couchjs binary
>> that has custom HTTP extensions to make the xhr work. At this point I =
don't think it
>> is reliable enough to mimic browser behaviour and that we shouldn't =
use it for vetting
>> the state of the code.
>=20
> This is likely true, but in this particular case I think there's a bug =
in the changes code (that I'm trying to dig out). It's nice that it =
works on your machine but on my machine, using FF, it fails often =
enough. Moreover it's been around for a long time now so I figure it's =
worth figuring out.=20
>=20
> I don't have a dog in this fight (.ie. a paying customer) so this =
failure doesn't bother me. With respect to policy, given the various =
bogocities of browsers, I'd recommend something like these CLI tests =
plus the etaps ought to be the "official"  tests for vetting, and part =
of the build

Not that I disagree, but part (most?) of the appeal of the browser based =
tests are that they run in a real-world client instead of the lab that =
is couchjs+http :)

Cheers
Jan
--

>=20
>=20
>>=20
>> It is very useful when developing new code to not have to switch to =
and reload the
>> browser over and over again.
>>=20
>> Cheers
>> Jan
>> --
>>=20
>>=20
>>=20
>>=20
>>>=20
>>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote:
>>>=20
>>>>=20
>>>> On 21 Mar 2010, at 06:04, Robert Dionne wrote:
>>>>=20
>>>>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote:
>>>>>=20
>>>>>>=20
>>>>>> On 20 Mar 2010, at 20:06, Paul Davis wrote:
>>>>>>=20
>>>>>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater <nslater@me.com> =
wrote:
>>>>>>>> I think faulty test case should block the release, if I am to =
have any
>>>>>>>> future sanity preparing releases. I don't want to delay and =
longer, so if
>>>>>>>> you guys are absolutely sure this is a test error and not code =
error, then I
>>>>>>>> propose that the test be commented out. Our tests form a =
contract between
>>>>>>>> us, internally, and our users. If that contract has a bug, it =
should be
>>>>>>>> removed or fixed - or it simply dilutes the importance of =
contract. If some
>>>>>>>> one comments out the test, and we agree it is not indicative of =
an important
>>>>>>>> bug, I can call the vote within hours.
>>>>>>>>=20
>>>>>>>=20
>>>>>>> I'd have to agree on this. =46rom the point of view of a =
release, if a
>>>>>>> test reports a failure then it should be made to not report a =
failure.
>>>>>>> If that's accomplished by disabling it, then there will be a =
commit
>>>>>>> with a message that explains why it was disabled and etc and =
such on
>>>>>>> and so forth.
>>>>>>=20
>>>>>> I'd do that if the test was failing for me :)
>>>>>=20
>>>>> it's not failing for you when you run changes.js with the CLI ?  =
Fails for me every time.=20
>>>>=20
>>>> I don't consider the CLI tests as part of the official test suite =
just yet.
>>>>=20
>>>> Cheers
>>>> Jan
>>>> --
>>>>=20
>>>>>=20
>>>>> Anyway I poked at this a bit yesterday and am not 100% sure the =
issue is in the test. I tried putting a sleep in with no luck. If my =
understanding of the JS is correct, CouchDB is supposed to be =
synchronous so it's not timing.
>>>>>=20
>>>>> If someone could comment on the test itself it would be helpful. =
The section of the code that fails:
>>>>>=20
>>>>> // changes get all_docs style with deleted docs
>>>>> var doc =3D {a:1};
>>>>> db.save(doc);
>>>>> db.deleteDoc(doc);
>>>>> var req =3D CouchDB.request("GET",=20
>>>>> =
"/test_suite_db/_changes?filter=3Dchanges_filter/bop&style=3Dall_docs");
>>>>> var resp =3D JSON.parse(req.responseText);
>>>>> TEquals(3, resp.results.length, "should return matching rows");
>>>>>=20
>>>>>=20
>>>>> seems odd to me. all_docs as I read the code will return docs with =
deletes and conflicts but in this call the filter bop will not apply to =
the doc {a:1} so I'm not sure what this delete prior to the call is =
about. Anyway I can make it fail in the debugger so perhaps I can find =
the root cause.
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>>=20
>>>>>> Cheers
>>>>>> Jan
>>>>>> --
>>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>>=20
>=20