Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAJ_m3YApvok8U-FAzTCz4+rQK6p2RRefN4F_MqFpiF-Fr8ioog@mail.gmail.com>
References: 
 <CALpjNyk1C28d4xTVrpGyObKzYQeKKysDLAeFrn4y-cMR43GawQ@mail.gmail.com>
	<CAJ_m3YCeDrBXysNNWuPqzT0Qik=XDR3dnzOADRXT_VkJu+G-zA@mail.gmail.com>
	<CALpjNy=-wjJRMVn-V6E4TsfoG4Gu=jQpJ8rm4zJxJZ0eKvxR9A@mail.gmail.com>
	<00EFB465-11C2-4023-ACEA-F80D3E3E5CB2@projectmastermind.com>
	<CAAL6JQiU5przo8Drh81=oRSWDYLvff=7cUv4X+nEh9WchB9-hw@mail.gmail.com>
	<CAJ_m3YAzspfw_6AMwsSDVG7LT3O0Z0xqOvo4Qz_=LxyRSVE8wQ@mail.gmail.com>
	<CAAL6JQgof67dc4QsBAotZc=meKUPmn+F8sKi26=mXDsFW_yBmQ@mail.gmail.com>
	<CAJ_m3YApvok8U-FAzTCz4+rQK6p2RRefN4F_MqFpiF-Fr8ioog@mail.gmail.com>
Date: Thu, 20 Oct 2011 21:25:28 +0100
Message-ID: 
 <CABvT1DFo4moex+iNnjS4-Tcww2ap-WkBWT5O6bxJff8rqVyMVg@mail.gmail.com>
Subject: Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
From: Robert Newson <rnewson@apache.org>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'll also note that the bug that killed round 1 of 1.1.1 was not found
by any test we currently have. All it would have taken is a test that
did any map call followed by almost any other bit of javascript (and
sm 1.7.0).

On 20 October 2011 21:22, Paul Davis <paul.joseph.davis@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <randall.leeds@gmail.com> =
wrote:
>> On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com>w=
rote:
>>
>>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org> wro=
te:
>>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>>> lee@projectmastermind.com>wrote:
>>> >
>>> >>
>>> >> > For what it's worth, a CLI based test system is what I was imagini=
ng
>>> >> > as well. Take Futon out of the mix and test CouchDB.
>>> >>
>>> >> IMO, If CouchDB is intended to be a server that can be accessed from
>>> >> the browser directly, then there should continue to be some kind of
>>> >> browser-based test suite that would serve to confirm this capability=
.
>>> >>
>>> >>
>>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
>>> >> several days, with the idea that I might begin to clean them up a bi=
t
>>> >> as time permits.
>>> >>
>>> >> I have found that, while some of these test failures are totally bog=
us,
>>> >> *some* of them actually do stem from real issues -- minor
>>> >> incompatibilities between CouchDB's http interface, and the internal
>>> >> mechanisms of modern browsers (XHR, caching, etc).
>>> >>
>>> >> These are problems that we're not going to catch with a stateless,
>>> >> cache-less http client running on the CLI. =A0(I can provide example=
s)
>>> >>
>>> >> These issues have the potential to cause real problems for
>>> >> developers of real browser-based apps "in the wild". =A0That means,
>>> >> there's valuable info to be gathered from the browser tests, Iff we
>>> >> can clean them up, and make them behave consistently; so that
>>> >> when they fail or succeed, we can actually trust the results.
>>> >>
>>> >>
>>> >> After digging around a good bit, I can see no reason why the existin=
g
>>> >> tests couldn't be cleaned up and made to work correctly in all curre=
nt
>>> >> versions of major browsers. =A0I also see no reason why the same tes=
ts
>>> >> couldn't be used successfully from the CLI and `make check` as well.
>>> >>
>>> >> I do see significant benefits to using the same javascript test code=
 in
>>> >> all environments we test.
>>> >>
>>> >> -Lee
>>> >> (irc: coltr)
>>> >>
>>> >
>>> > =A0+1
>>> > Verify Installation could grow into a suite of browser/futon tests th=
at
>>> > verify that futon (and apps in general) work, including interactions =
with
>>> > proxies and the like.
>>>
>>> Sure. Client tests that test the client are fine.
>>>
>>> > The test suite for developers should run cleanly from the CLI as part=
 of
>>> > make check, but continue to be exposed in futon. We should work to be
>>> sure
>>> > they function as well as possible, for the reasons you provide.
>>> >
>>>
>>> Blargh no. Server tests should be testing the server. The entire point
>>> of moving to the command line is so that we don't have to maintain the
>>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>>> release thread) and the wildly varying reports of test output. The
>>> situation is just a waste of time for everyone involved.
>>>
>>> > I think the JS testing situation is a great place for people to jump =
in
>>> and
>>> > help out, especially with the browser environment diversity.
>>> >
>>>
>>> Sure, but I don't see what this has to do with browsers.
>>>
>>
>> People who aren't into the internals can help to fix the suite to work i=
n
>> different browser environments. That's all I meant.
>>
>
> Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
> "If these people exist, why do I not see anything in JIRA?"
>
>> I suggested that the CLI tests be exposed in Futon because I think there=
 are
>> probably some JS heads in this community who wouldn't have too much trou=
ble
>> fixing a lot of the user agent related issues in the test suite. I didn'=
t
>> mean to suggest that it should continue to be part of the release proced=
ure
>> (it shouldn't) or that we should feel 100% obligated to make sure they p=
ass
>> in 100% of environments (we can't and shouldn't), but J. Lee's point abo=
ut
>> how keeping such tests around can sometimes expose interesting problems =
we
>> wouldn't otherwise see, possible outside the CouchDB codebase even, is
>> worthwhile.
>>
>> -Randall
>>
>
> We've had these tests for three years or more now. Perhaps I'm just
> being dense today but I can't think of a single specific case where
> testing things in the browser has lead to a bug report/fix that we
> wouldn't have found with pure CLI tests.
>
> The only thing that I'm aware that the tests have done for us is
> required us to exert a nontrivial amount of effort to keep them
> running in multiple browser environments. I have no interest in
> maintaing these as tests runnable in the browser. I want to create a
> CLI test environment that promotes stable, repeatable, concise tests.
> Running these in a browser is the antithesis to such an environment.
>