From dev-return-18672-apmail-couchdb-dev-archive=couchdb.apache.org@couchdb.apache.org Thu Oct 20 20:23:19 2011 Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E90FF7B7F for ; Thu, 20 Oct 2011 20:23:19 +0000 (UTC) Received: (qmail 44900 invoked by uid 500); 20 Oct 2011 20:23:19 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 44856 invoked by uid 500); 20 Oct 2011 20:23:19 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 44848 invoked by uid 99); 20 Oct 2011 20:23:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 20:23:19 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com designates 209.85.220.180 as permitted sender) Received: from [209.85.220.180] (HELO mail-vx0-f180.google.com) (209.85.220.180) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 20:23:13 +0000 Received: by vcbfo11 with SMTP id fo11so4144397vcb.11 for ; Thu, 20 Oct 2011 13:22:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=BJocoZ+MVB3oZRk3Z8eoFZPGwHJxUiJUdXBM6etEMys=; b=G/j6T3sTsxSbV4qwtNG6N6AlJXo+rRn1i0YxaedWnseJ/Yjzp1jUF+45LFhhIKX0Ws zKw43coKh04ok7bIsgKUFyX0BNp+iT/R+biz2YQmul5IKITiGvCKHkveHQbuTpABXcC/ iuleQFbrxq+0FOkFcUz8THj3xZ72pvBHcf+Yo= Received: by 10.52.72.104 with SMTP id c8mr3459515vdv.105.1319142172093; Thu, 20 Oct 2011 13:22:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.167.71 with HTTP; Thu, 20 Oct 2011 13:22:12 -0700 (PDT) In-Reply-To: References: <00EFB465-11C2-4023-ACEA-F80D3E3E5CB2@projectmastermind.com> From: Paul Davis Date: Thu, 20 Oct 2011 15:22:12 -0500 Message-ID: Subject: Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release) To: dev@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds wr= ote: > On Thu, Oct 20, 2011 at 13:42, Paul Davis wr= ote: > >> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds wrot= e: >> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane < >> lee@projectmastermind.com>wrote: >> > >> >> >> >> > For what it's worth, a CLI based test system is what I was imaginin= g >> >> > as well. Take Futon out of the mix and test CouchDB. >> >> >> >> IMO, If CouchDB is intended to be a server that can be accessed from >> >> the browser directly, then there should continue to be some kind of >> >> browser-based test suite that would serve to confirm this capability. >> >> >> >> >> >> I have been looking closely at the Futon tests in 1.1.0 for the last >> >> several days, with the idea that I might begin to clean them up a bit >> >> as time permits. >> >> >> >> I have found that, while some of these test failures are totally bogu= s, >> >> *some* of them actually do stem from real issues -- minor >> >> incompatibilities between CouchDB's http interface, and the internal >> >> mechanisms of modern browsers (XHR, caching, etc). >> >> >> >> These are problems that we're not going to catch with a stateless, >> >> cache-less http client running on the CLI. =A0(I can provide examples= ) >> >> >> >> These issues have the potential to cause real problems for >> >> developers of real browser-based apps "in the wild". =A0That means, >> >> there's valuable info to be gathered from the browser tests, Iff we >> >> can clean them up, and make them behave consistently; so that >> >> when they fail or succeed, we can actually trust the results. >> >> >> >> >> >> After digging around a good bit, I can see no reason why the existing >> >> tests couldn't be cleaned up and made to work correctly in all curren= t >> >> versions of major browsers. =A0I also see no reason why the same test= s >> >> couldn't be used successfully from the CLI and `make check` as well. >> >> >> >> I do see significant benefits to using the same javascript test code = in >> >> all environments we test. >> >> >> >> -Lee >> >> (irc: coltr) >> >> >> > >> > =A0+1 >> > Verify Installation could grow into a suite of browser/futon tests tha= t >> > verify that futon (and apps in general) work, including interactions w= ith >> > proxies and the like. >> >> Sure. Client tests that test the client are fine. >> >> > The test suite for developers should run cleanly from the CLI as part = of >> > make check, but continue to be exposed in futon. We should work to be >> sure >> > they function as well as possible, for the reasons you provide. >> > >> >> Blargh no. Server tests should be testing the server. The entire point >> of moving to the command line is so that we don't have to maintain the >> Futon test suite. Just look at the 1.1.1 thread (or damn near any >> release thread) and the wildly varying reports of test output. The >> situation is just a waste of time for everyone involved. >> >> > I think the JS testing situation is a great place for people to jump i= n >> and >> > help out, especially with the browser environment diversity. >> > >> >> Sure, but I don't see what this has to do with browsers. >> > > People who aren't into the internals can help to fix the suite to work in > different browser environments. That's all I meant. > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically, "If these people exist, why do I not see anything in JIRA?" > I suggested that the CLI tests be exposed in Futon because I think there = are > probably some JS heads in this community who wouldn't have too much troub= le > fixing a lot of the user agent related issues in the test suite. I didn't > mean to suggest that it should continue to be part of the release procedu= re > (it shouldn't) or that we should feel 100% obligated to make sure they pa= ss > in 100% of environments (we can't and shouldn't), but J. Lee's point abou= t > how keeping such tests around can sometimes expose interesting problems w= e > wouldn't otherwise see, possible outside the CouchDB codebase even, is > worthwhile. > > -Randall > We've had these tests for three years or more now. Perhaps I'm just being dense today but I can't think of a single specific case where testing things in the browser has lead to a bug report/fix that we wouldn't have found with pure CLI tests. The only thing that I'm aware that the tests have done for us is required us to exert a nontrivial amount of effort to keep them running in multiple browser environments. I have no interest in maintaing these as tests runnable in the browser. I want to create a CLI test environment that promotes stable, repeatable, concise tests. Running these in a browser is the antithesis to such an environment.