couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randall Leeds <randall.le...@gmail.com>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Thu, 20 Oct 2011 18:45:44 GMT
On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com>wrote:

> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org> wrote:
> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
> lee@projectmastermind.com>wrote:
> >
> >>
> >> > For what it's worth, a CLI based test system is what I was imagining
> >> > as well. Take Futon out of the mix and test CouchDB.
> >>
> >> IMO, If CouchDB is intended to be a server that can be accessed from
> >> the browser directly, then there should continue to be some kind of
> >> browser-based test suite that would serve to confirm this capability.
> >>
> >>
> >> I have been looking closely at the Futon tests in 1.1.0 for the last
> >> several days, with the idea that I might begin to clean them up a bit
> >> as time permits.
> >>
> >> I have found that, while some of these test failures are totally bogus,
> >> *some* of them actually do stem from real issues -- minor
> >> incompatibilities between CouchDB's http interface, and the internal
> >> mechanisms of modern browsers (XHR, caching, etc).
> >>
> >> These are problems that we're not going to catch with a stateless,
> >> cache-less http client running on the CLI.  (I can provide examples)
> >>
> >> These issues have the potential to cause real problems for
> >> developers of real browser-based apps "in the wild".  That means,
> >> there's valuable info to be gathered from the browser tests, Iff we
> >> can clean them up, and make them behave consistently; so that
> >> when they fail or succeed, we can actually trust the results.
> >>
> >>
> >> After digging around a good bit, I can see no reason why the existing
> >> tests couldn't be cleaned up and made to work correctly in all current
> >> versions of major browsers.  I also see no reason why the same tests
> >> couldn't be used successfully from the CLI and `make check` as well.
> >>
> >> I do see significant benefits to using the same javascript test code in
> >> all environments we test.
> >>
> >> -Lee
> >> (irc: coltr)
> >>
> >
> >  +1
> > Verify Installation could grow into a suite of browser/futon tests that
> > verify that futon (and apps in general) work, including interactions with
> > proxies and the like.
>
> Sure. Client tests that test the client are fine.
>
> > The test suite for developers should run cleanly from the CLI as part of
> > make check, but continue to be exposed in futon. We should work to be
> sure
> > they function as well as possible, for the reasons you provide.
> >
>
> Blargh no. Server tests should be testing the server. The entire point
> of moving to the command line is so that we don't have to maintain the
> Futon test suite. Just look at the 1.1.1 thread (or damn near any
> release thread) and the wildly varying reports of test output. The
> situation is just a waste of time for everyone involved.
>
> > I think the JS testing situation is a great place for people to jump in
> and
> > help out, especially with the browser environment diversity.
> >
>
> Sure, but I don't see what this has to do with browsers.
>

People who aren't into the internals can help to fix the suite to work in
different browser environments. That's all I meant.

I suggested that the CLI tests be exposed in Futon because I think there are
probably some JS heads in this community who wouldn't have too much trouble
fixing a lot of the user agent related issues in the test suite. I didn't
mean to suggest that it should continue to be part of the release procedure
(it shouldn't) or that we should feel 100% obligated to make sure they pass
in 100% of environments (we can't and shouldn't), but J. Lee's point about
how keeping such tests around can sometimes expose interesting problems we
wouldn't otherwise see, possible outside the CouchDB codebase even, is
worthwhile.

-Randall

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message