couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Fri, 21 Oct 2011 04:35:04 GMT
Honestly, I was looking for the video of the one lady screaming to
provide commentary on they "I agree with what he said" comment. But I
failed slightly, but only slightly enough to illicit an awesome
introspection on a random Beatles video.

On Thu, Oct 20, 2011 at 11:17 PM, Noah Slater <nslater@tumbolia.org> wrote:
> Wait, did you post that because me and you rock this project like the
> Beatles rocked America? Or did you post it with the intention of the song
> She Loves You being a sort of meta-commentary on our enviable, and now
> infamous, bromance?
>
> On Fri, Oct 21, 2011 at 5:13 AM, Paul Davis <paul.joseph.davis@gmail.com>wrote:
>
>> http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s
>>
>> On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <nslater@tumbolia.org> wrote:
>> > +1 on all the stuff Paul said.
>> >
>> > On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rnewson@apache.org>
>> wrote:
>> >
>> >> I'll also note that the bug that killed round 1 of 1.1.1 was not found
>> >> by any test we currently have. All it would have taken is a test that
>> >> did any map call followed by almost any other bit of javascript (and
>> >> sm 1.7.0).
>> >>
>> >> On 20 October 2011 21:22, Paul Davis <paul.joseph.davis@gmail.com>
>> wrote:
>> >> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <
>> randall.leeds@gmail.com>
>> >> wrote:
>> >> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <
>> paul.joseph.davis@gmail.com
>> >> >wrote:
>> >> >>
>> >> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org>
>> >> wrote:
>> >> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> >> >>> lee@projectmastermind.com>wrote:
>> >> >>> >
>> >> >>> >>
>> >> >>> >> > For what it's worth, a CLI based test system
is what I was
>> >> imagining
>> >> >>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >> >>> >>
>> >> >>> >> IMO, If CouchDB is intended to be a server that can
be accessed
>> from
>> >> >>> >> the browser directly, then there should continue to
be some kind
>> of
>> >> >>> >> browser-based test suite that would serve to confirm
this
>> >> capability.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> I have been looking closely at the Futon tests in
1.1.0 for the
>> last
>> >> >>> >> several days, with the idea that I might begin to
clean them up a
>> >> bit
>> >> >>> >> as time permits.
>> >> >>> >>
>> >> >>> >> I have found that, while some of these test failures
are totally
>> >> bogus,
>> >> >>> >> *some* of them actually do stem from real issues --
minor
>> >> >>> >> incompatibilities between CouchDB's http interface,
and the
>> internal
>> >> >>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >> >>> >>
>> >> >>> >> These are problems that we're not going to catch with
a
>> stateless,
>> >> >>> >> cache-less http client running on the CLI.  (I can
provide
>> examples)
>> >> >>> >>
>> >> >>> >> These issues have the potential to cause real problems
for
>> >> >>> >> developers of real browser-based apps "in the wild".
 That means,
>> >> >>> >> there's valuable info to be gathered from the browser
tests, Iff
>> we
>> >> >>> >> can clean them up, and make them behave consistently;
so that
>> >> >>> >> when they fail or succeed, we can actually trust the
results.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> After digging around a good bit, I can see no reason
why the
>> >> existing
>> >> >>> >> tests couldn't be cleaned up and made to work correctly
in all
>> >> current
>> >> >>> >> versions of major browsers.  I also see no reason
why the same
>> tests
>> >> >>> >> couldn't be used successfully from the CLI and `make
check` as
>> well.
>> >> >>> >>
>> >> >>> >> I do see significant benefits to using the same javascript
test
>> code
>> >> in
>> >> >>> >> all environments we test.
>> >> >>> >>
>> >> >>> >> -Lee
>> >> >>> >> (irc: coltr)
>> >> >>> >>
>> >> >>> >
>> >> >>> >  +1
>> >> >>> > Verify Installation could grow into a suite of browser/futon
tests
>> >> that
>> >> >>> > verify that futon (and apps in general) work, including
>> interactions
>> >> with
>> >> >>> > proxies and the like.
>> >> >>>
>> >> >>> Sure. Client tests that test the client are fine.
>> >> >>>
>> >> >>> > The test suite for developers should run cleanly from
the CLI as
>> part
>> >> of
>> >> >>> > make check, but continue to be exposed in futon. We should
work to
>> be
>> >> >>> sure
>> >> >>> > they function as well as possible, for the reasons you
provide.
>> >> >>> >
>> >> >>>
>> >> >>> Blargh no. Server tests should be testing the server. The entire
>> point
>> >> >>> of moving to the command line is so that we don't have to maintain
>> the
>> >> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near
any
>> >> >>> release thread) and the wildly varying reports of test output.
The
>> >> >>> situation is just a waste of time for everyone involved.
>> >> >>>
>> >> >>> > I think the JS testing situation is a great place for
people to
>> jump
>> >> in
>> >> >>> and
>> >> >>> > help out, especially with the browser environment diversity.
>> >> >>> >
>> >> >>>
>> >> >>> Sure, but I don't see what this has to do with browsers.
>> >> >>>
>> >> >>
>> >> >> People who aren't into the internals can help to fix the suite
to
>> work
>> >> in
>> >> >> different browser environments. That's all I meant.
>> >> >>
>> >> >
>> >> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
>> >> > "If these people exist, why do I not see anything in JIRA?"
>> >> >
>> >> >> I suggested that the CLI tests be exposed in Futon because I think
>> there
>> >> are
>> >> >> probably some JS heads in this community who wouldn't have too
much
>> >> trouble
>> >> >> fixing a lot of the user agent related issues in the test suite.
I
>> >> didn't
>> >> >> mean to suggest that it should continue to be part of the release
>> >> procedure
>> >> >> (it shouldn't) or that we should feel 100% obligated to make sure
>> they
>> >> pass
>> >> >> in 100% of environments (we can't and shouldn't), but J. Lee's
point
>> >> about
>> >> >> how keeping such tests around can sometimes expose interesting
>> problems
>> >> we
>> >> >> wouldn't otherwise see, possible outside the CouchDB codebase even,
>> is
>> >> >> worthwhile.
>> >> >>
>> >> >> -Randall
>> >> >>
>> >> >
>> >> > We've had these tests for three years or more now. Perhaps I'm just
>> >> > being dense today but I can't think of a single specific case where
>> >> > testing things in the browser has lead to a bug report/fix that we
>> >> > wouldn't have found with pure CLI tests.
>> >> >
>> >> > The only thing that I'm aware that the tests have done for us is
>> >> > required us to exert a nontrivial amount of effort to keep them
>> >> > running in multiple browser environments. I have no interest in
>> >> > maintaing these as tests runnable in the browser. I want to create
a
>> >> > CLI test environment that promotes stable, repeatable, concise tests.
>> >> > Running these in a browser is the antithesis to such an environment.
>> >> >
>> >>
>> >
>>
>

Mime
View raw message