couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Fri, 21 Oct 2011 04:13:28 GMT
http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s

On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <nslater@tumbolia.org> wrote:
> +1 on all the stuff Paul said.
>
> On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rnewson@apache.org> wrote:
>
>> I'll also note that the bug that killed round 1 of 1.1.1 was not found
>> by any test we currently have. All it would have taken is a test that
>> did any map call followed by almost any other bit of javascript (and
>> sm 1.7.0).
>>
>> On 20 October 2011 21:22, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <randall.leeds@gmail.com>
>> wrote:
>> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com
>> >wrote:
>> >>
>> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org>
>> wrote:
>> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> >>> lee@projectmastermind.com>wrote:
>> >>> >
>> >>> >>
>> >>> >> > For what it's worth, a CLI based test system is what I
was
>> imagining
>> >>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >>> >>
>> >>> >> IMO, If CouchDB is intended to be a server that can be accessed
from
>> >>> >> the browser directly, then there should continue to be some
kind of
>> >>> >> browser-based test suite that would serve to confirm this
>> capability.
>> >>> >>
>> >>> >>
>> >>> >> I have been looking closely at the Futon tests in 1.1.0 for
the last
>> >>> >> several days, with the idea that I might begin to clean them
up a
>> bit
>> >>> >> as time permits.
>> >>> >>
>> >>> >> I have found that, while some of these test failures are totally
>> bogus,
>> >>> >> *some* of them actually do stem from real issues -- minor
>> >>> >> incompatibilities between CouchDB's http interface, and the
internal
>> >>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >>> >>
>> >>> >> These are problems that we're not going to catch with a stateless,
>> >>> >> cache-less http client running on the CLI.  (I can provide
examples)
>> >>> >>
>> >>> >> These issues have the potential to cause real problems for
>> >>> >> developers of real browser-based apps "in the wild".  That
means,
>> >>> >> there's valuable info to be gathered from the browser tests,
Iff we
>> >>> >> can clean them up, and make them behave consistently; so that
>> >>> >> when they fail or succeed, we can actually trust the results.
>> >>> >>
>> >>> >>
>> >>> >> After digging around a good bit, I can see no reason why the
>> existing
>> >>> >> tests couldn't be cleaned up and made to work correctly in
all
>> current
>> >>> >> versions of major browsers.  I also see no reason why the
same tests
>> >>> >> couldn't be used successfully from the CLI and `make check`
as well.
>> >>> >>
>> >>> >> I do see significant benefits to using the same javascript
test code
>> in
>> >>> >> all environments we test.
>> >>> >>
>> >>> >> -Lee
>> >>> >> (irc: coltr)
>> >>> >>
>> >>> >
>> >>> >  +1
>> >>> > Verify Installation could grow into a suite of browser/futon tests
>> that
>> >>> > verify that futon (and apps in general) work, including interactions
>> with
>> >>> > proxies and the like.
>> >>>
>> >>> Sure. Client tests that test the client are fine.
>> >>>
>> >>> > The test suite for developers should run cleanly from the CLI as
part
>> of
>> >>> > make check, but continue to be exposed in futon. We should work
to be
>> >>> sure
>> >>> > they function as well as possible, for the reasons you provide.
>> >>> >
>> >>>
>> >>> Blargh no. Server tests should be testing the server. The entire point
>> >>> of moving to the command line is so that we don't have to maintain the
>> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>> >>> release thread) and the wildly varying reports of test output. The
>> >>> situation is just a waste of time for everyone involved.
>> >>>
>> >>> > I think the JS testing situation is a great place for people to
jump
>> in
>> >>> and
>> >>> > help out, especially with the browser environment diversity.
>> >>> >
>> >>>
>> >>> Sure, but I don't see what this has to do with browsers.
>> >>>
>> >>
>> >> People who aren't into the internals can help to fix the suite to work
>> in
>> >> different browser environments. That's all I meant.
>> >>
>> >
>> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
>> > "If these people exist, why do I not see anything in JIRA?"
>> >
>> >> I suggested that the CLI tests be exposed in Futon because I think there
>> are
>> >> probably some JS heads in this community who wouldn't have too much
>> trouble
>> >> fixing a lot of the user agent related issues in the test suite. I
>> didn't
>> >> mean to suggest that it should continue to be part of the release
>> procedure
>> >> (it shouldn't) or that we should feel 100% obligated to make sure they
>> pass
>> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
>> about
>> >> how keeping such tests around can sometimes expose interesting problems
>> we
>> >> wouldn't otherwise see, possible outside the CouchDB codebase even, is
>> >> worthwhile.
>> >>
>> >> -Randall
>> >>
>> >
>> > We've had these tests for three years or more now. Perhaps I'm just
>> > being dense today but I can't think of a single specific case where
>> > testing things in the browser has lead to a bug report/fix that we
>> > wouldn't have found with pure CLI tests.
>> >
>> > The only thing that I'm aware that the tests have done for us is
>> > required us to exert a nontrivial amount of effort to keep them
>> > running in multiple browser environments. I have no interest in
>> > maintaing these as tests runnable in the browser. I want to create a
>> > CLI test environment that promotes stable, repeatable, concise tests.
>> > Running these in a browser is the antithesis to such an environment.
>> >
>>
>

Mime
View raw message