Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 344F19858 for ; Fri, 21 Oct 2011 04:36:15 +0000 (UTC) Received: (qmail 48977 invoked by uid 500); 21 Oct 2011 04:36:14 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 48946 invoked by uid 500); 21 Oct 2011 04:36:13 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 48933 invoked by uid 99); 21 Oct 2011 04:36:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 04:36:11 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com designates 209.85.212.52 as permitted sender) Received: from [209.85.212.52] (HELO mail-vw0-f52.google.com) (209.85.212.52) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2011 04:36:05 +0000 Received: by vws10 with SMTP id 10so3872130vws.11 for ; Thu, 20 Oct 2011 21:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=jWY0ldq7RV/yjCswysEDBEAItV9w6+3oq3Z5st+CMkA=; b=aYM8a6kTo+4S8jWY7BMVRne8yAD7LeZkQ81QuiCwYlzUTvtMJRb83vxV9Y9lyA1Fv7 BU+0NjuD2ZFDvbdUEZhA/LlYXPEwm5JNOLSyunWD4f+hf/2UksfLL7wh/BwdlJRu5mQN 3pT68LbE+qQY+W/PX/gg33VM15WsSsbcMaTrE= Received: by 10.52.71.200 with SMTP id x8mr13153363vdu.54.1319171744069; Thu, 20 Oct 2011 21:35:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.167.71 with HTTP; Thu, 20 Oct 2011 21:35:04 -0700 (PDT) In-Reply-To: References: <00EFB465-11C2-4023-ACEA-F80D3E3E5CB2@projectmastermind.com> From: Paul Davis Date: Thu, 20 Oct 2011 23:35:04 -0500 Message-ID: Subject: Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release) To: dev@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Honestly, I was looking for the video of the one lady screaming to provide commentary on they "I agree with what he said" comment. But I failed slightly, but only slightly enough to illicit an awesome introspection on a random Beatles video. On Thu, Oct 20, 2011 at 11:17 PM, Noah Slater wrote: > Wait, did you post that because me and you rock this project like the > Beatles rocked America? Or did you post it with the intention of the song > She Loves You being a sort of meta-commentary on our enviable, and now > infamous, bromance? > > On Fri, Oct 21, 2011 at 5:13 AM, Paul Davis = wrote: > >> http://www.youtube.com/watch?feature=3Dplayer_detailpage&v=3DG6j5bve7O5E= #t=3D109s >> >> On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater wrot= e: >> > +1 on all the stuff Paul said. >> > >> > On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson >> wrote: >> > >> >> I'll also note that the bug that killed round 1 of 1.1.1 was not foun= d >> >> by any test we currently have. All it would have taken is a test that >> >> did any map call followed by almost any other bit of javascript (and >> >> sm 1.7.0). >> >> >> >> On 20 October 2011 21:22, Paul Davis >> wrote: >> >> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds < >> randall.leeds@gmail.com> >> >> wrote: >> >> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis < >> paul.joseph.davis@gmail.com >> >> >wrote: >> >> >> >> >> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds >> >> wrote: >> >> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane < >> >> >>> lee@projectmastermind.com>wrote: >> >> >>> > >> >> >>> >> >> >> >>> >> > For what it's worth, a CLI based test system is what I was >> >> imagining >> >> >>> >> > as well. Take Futon out of the mix and test CouchDB. >> >> >>> >> >> >> >>> >> IMO, If CouchDB is intended to be a server that can be accesse= d >> from >> >> >>> >> the browser directly, then there should continue to be some ki= nd >> of >> >> >>> >> browser-based test suite that would serve to confirm this >> >> capability. >> >> >>> >> >> >> >>> >> >> >> >>> >> I have been looking closely at the Futon tests in 1.1.0 for th= e >> last >> >> >>> >> several days, with the idea that I might begin to clean them u= p a >> >> bit >> >> >>> >> as time permits. >> >> >>> >> >> >> >>> >> I have found that, while some of these test failures are total= ly >> >> bogus, >> >> >>> >> *some* of them actually do stem from real issues -- minor >> >> >>> >> incompatibilities between CouchDB's http interface, and the >> internal >> >> >>> >> mechanisms of modern browsers (XHR, caching, etc). >> >> >>> >> >> >> >>> >> These are problems that we're not going to catch with a >> stateless, >> >> >>> >> cache-less http client running on the CLI. =A0(I can provide >> examples) >> >> >>> >> >> >> >>> >> These issues have the potential to cause real problems for >> >> >>> >> developers of real browser-based apps "in the wild". =A0That m= eans, >> >> >>> >> there's valuable info to be gathered from the browser tests, I= ff >> we >> >> >>> >> can clean them up, and make them behave consistently; so that >> >> >>> >> when they fail or succeed, we can actually trust the results. >> >> >>> >> >> >> >>> >> >> >> >>> >> After digging around a good bit, I can see no reason why the >> >> existing >> >> >>> >> tests couldn't be cleaned up and made to work correctly in all >> >> current >> >> >>> >> versions of major browsers. =A0I also see no reason why the sa= me >> tests >> >> >>> >> couldn't be used successfully from the CLI and `make check` as >> well. >> >> >>> >> >> >> >>> >> I do see significant benefits to using the same javascript tes= t >> code >> >> in >> >> >>> >> all environments we test. >> >> >>> >> >> >> >>> >> -Lee >> >> >>> >> (irc: coltr) >> >> >>> >> >> >> >>> > >> >> >>> > =A0+1 >> >> >>> > Verify Installation could grow into a suite of browser/futon te= sts >> >> that >> >> >>> > verify that futon (and apps in general) work, including >> interactions >> >> with >> >> >>> > proxies and the like. >> >> >>> >> >> >>> Sure. Client tests that test the client are fine. >> >> >>> >> >> >>> > The test suite for developers should run cleanly from the CLI a= s >> part >> >> of >> >> >>> > make check, but continue to be exposed in futon. We should work= to >> be >> >> >>> sure >> >> >>> > they function as well as possible, for the reasons you provide. >> >> >>> > >> >> >>> >> >> >>> Blargh no. Server tests should be testing the server. The entire >> point >> >> >>> of moving to the command line is so that we don't have to maintai= n >> the >> >> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any >> >> >>> release thread) and the wildly varying reports of test output. Th= e >> >> >>> situation is just a waste of time for everyone involved. >> >> >>> >> >> >>> > I think the JS testing situation is a great place for people to >> jump >> >> in >> >> >>> and >> >> >>> > help out, especially with the browser environment diversity. >> >> >>> > >> >> >>> >> >> >>> Sure, but I don't see what this has to do with browsers. >> >> >>> >> >> >> >> >> >> People who aren't into the internals can help to fix the suite to >> work >> >> in >> >> >> different browser environments. That's all I meant. >> >> >> >> >> > >> >> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorical= ly, >> >> > "If these people exist, why do I not see anything in JIRA?" >> >> > >> >> >> I suggested that the CLI tests be exposed in Futon because I think >> there >> >> are >> >> >> probably some JS heads in this community who wouldn't have too muc= h >> >> trouble >> >> >> fixing a lot of the user agent related issues in the test suite. I >> >> didn't >> >> >> mean to suggest that it should continue to be part of the release >> >> procedure >> >> >> (it shouldn't) or that we should feel 100% obligated to make sure >> they >> >> pass >> >> >> in 100% of environments (we can't and shouldn't), but J. Lee's poi= nt >> >> about >> >> >> how keeping such tests around can sometimes expose interesting >> problems >> >> we >> >> >> wouldn't otherwise see, possible outside the CouchDB codebase even= , >> is >> >> >> worthwhile. >> >> >> >> >> >> -Randall >> >> >> >> >> > >> >> > We've had these tests for three years or more now. Perhaps I'm just >> >> > being dense today but I can't think of a single specific case where >> >> > testing things in the browser has lead to a bug report/fix that we >> >> > wouldn't have found with pure CLI tests. >> >> > >> >> > The only thing that I'm aware that the tests have done for us is >> >> > required us to exert a nontrivial amount of effort to keep them >> >> > running in multiple browser environments. I have no interest in >> >> > maintaing these as tests runnable in the browser. I want to create = a >> >> > CLI test environment that promotes stable, repeatable, concise test= s. >> >> > Running these in a browser is the antithesis to such an environment= . >> >> > >> >> >> > >> >