couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Thu, 20 Oct 2011 17:39:03 GMT
On Wed, Oct 19, 2011 at 10:38 PM, J. Lee Coltrane
<lee@projectmastermind.com> wrote:
>
>> For what it's worth, a CLI based test system is what I was imagining
>> as well. Take Futon out of the mix and test CouchDB.
>
> IMO, If CouchDB is intended to be a server that can be accessed from
> the browser directly, then there should continue to be some kind of
> browser-based test suite that would serve to confirm this capability.
>

CouchDB is intended to be a server that can be accessed from HTTP
clients. Browsers are but one of a huge range of clients. I do agree
that there should be a browser based test suite, but I'm proposing
that these browser based tests should be testing the browser and not
testing CouchDB internals.

>
> I have been looking closely at the Futon tests in 1.1.0 for the last
> several days, with the idea that I might begin to clean them up a bit
> as time permits.
>
> I have found that, while some of these test failures are totally bogus,
> *some* of them actually do stem from real issues -- minor
> incompatibilities between CouchDB's http interface, and the internal
> mechanisms of modern browsers (XHR, caching, etc).
>
> These are problems that we're not going to catch with a stateless,
> cache-less http client running on the CLI.  (I can provide examples)
>

And I can provide examples where having a stateful caching client
merely exists to confound the test code. The issue is that using a
stateful caching client means that all tests have to account for this,
even tests that have nothing to do with such things.

> These issues have the potential to cause real problems for
> developers of real browser-based apps "in the wild".  That means,
> there's valuable info to be gathered from the browser tests, Iff we
> can clean them up, and make them behave consistently; so that
> when they fail or succeed, we can actually trust the results.
>
>
> After digging around a good bit, I can see no reason why the existing
> tests couldn't be cleaned up and made to work correctly in all current
> versions of major browsers.  I also see no reason why the same tests
> couldn't be used successfully from the CLI and `make check` as well.
>

There are some tests that make use of XHR directly which are incapable
of being run from the CLI test runner for one. There are also issues
in differences of caching implementations. There are even differences
of caching against localhost vs a remote server. I consider every time
I've had to diagnose a difference in browser behavior as an example of
precisely why (most of) these tests do not belong in a browser. Not
only is this a waste of time, it merely serves to make the test suite
less trustable when an error occurs.

We can hand wave about cleaning up the tests to make them more
reliable, but that's ignoring the fact that we're running the test
suite in huge monolithic environments that have a decades long history
of maddeningly subtle different semantics.

> I do see significant benefits to using the same javascript test code in
> all environments we test.
>

What's the benefit to maintaing assertions in the browser about view
output according to the UCA. Or whether or not the database is leaking
file descriptors. Or couch_os_process is properly caching couchjs
processes?

> -Lee
> (irc: coltr)
>

I don't mean to be cranky at you directly, but I am quite tired of
dealing with browsers to test a database. It was a bad idea to start
with and I've been trying to argue for the change for years now.

Mime
View raw message