couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Thu, 20 Oct 2011 20:22:12 GMT
On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <randall.leeds@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com>wrote:
>
>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org> wrote:
>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> lee@projectmastermind.com>wrote:
>> >
>> >>
>> >> > For what it's worth, a CLI based test system is what I was imagining
>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >>
>> >> IMO, If CouchDB is intended to be a server that can be accessed from
>> >> the browser directly, then there should continue to be some kind of
>> >> browser-based test suite that would serve to confirm this capability.
>> >>
>> >>
>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
>> >> several days, with the idea that I might begin to clean them up a bit
>> >> as time permits.
>> >>
>> >> I have found that, while some of these test failures are totally bogus,
>> >> *some* of them actually do stem from real issues -- minor
>> >> incompatibilities between CouchDB's http interface, and the internal
>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >>
>> >> These are problems that we're not going to catch with a stateless,
>> >> cache-less http client running on the CLI.  (I can provide examples)
>> >>
>> >> These issues have the potential to cause real problems for
>> >> developers of real browser-based apps "in the wild".  That means,
>> >> there's valuable info to be gathered from the browser tests, Iff we
>> >> can clean them up, and make them behave consistently; so that
>> >> when they fail or succeed, we can actually trust the results.
>> >>
>> >>
>> >> After digging around a good bit, I can see no reason why the existing
>> >> tests couldn't be cleaned up and made to work correctly in all current
>> >> versions of major browsers.  I also see no reason why the same tests
>> >> couldn't be used successfully from the CLI and `make check` as well.
>> >>
>> >> I do see significant benefits to using the same javascript test code in
>> >> all environments we test.
>> >>
>> >> -Lee
>> >> (irc: coltr)
>> >>
>> >
>> >  +1
>> > Verify Installation could grow into a suite of browser/futon tests that
>> > verify that futon (and apps in general) work, including interactions with
>> > proxies and the like.
>>
>> Sure. Client tests that test the client are fine.
>>
>> > The test suite for developers should run cleanly from the CLI as part of
>> > make check, but continue to be exposed in futon. We should work to be
>> sure
>> > they function as well as possible, for the reasons you provide.
>> >
>>
>> Blargh no. Server tests should be testing the server. The entire point
>> of moving to the command line is so that we don't have to maintain the
>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>> release thread) and the wildly varying reports of test output. The
>> situation is just a waste of time for everyone involved.
>>
>> > I think the JS testing situation is a great place for people to jump in
>> and
>> > help out, especially with the browser environment diversity.
>> >
>>
>> Sure, but I don't see what this has to do with browsers.
>>
>
> People who aren't into the internals can help to fix the suite to work in
> different browser environments. That's all I meant.
>

Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
"If these people exist, why do I not see anything in JIRA?"

> I suggested that the CLI tests be exposed in Futon because I think there are
> probably some JS heads in this community who wouldn't have too much trouble
> fixing a lot of the user agent related issues in the test suite. I didn't
> mean to suggest that it should continue to be part of the release procedure
> (it shouldn't) or that we should feel 100% obligated to make sure they pass
> in 100% of environments (we can't and shouldn't), but J. Lee's point about
> how keeping such tests around can sometimes expose interesting problems we
> wouldn't otherwise see, possible outside the CouchDB codebase even, is
> worthwhile.
>
> -Randall
>

We've had these tests for three years or more now. Perhaps I'm just
being dense today but I can't think of a single specific case where
testing things in the browser has lead to a bug report/fix that we
wouldn't have found with pure CLI tests.

The only thing that I'm aware that the tests have done for us is
required us to exert a nontrivial amount of effort to keep them
running in multiple browser environments. I have no interest in
maintaing these as tests runnable in the browser. I want to create a
CLI test environment that promotes stable, repeatable, concise tests.
Running these in a browser is the antithesis to such an environment.

Mime
View raw message