couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@tumbolia.org>
Subject Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)
Date Fri, 21 Oct 2011 04:17:13 GMT
Wait, did you post that because me and you rock this project like the
Beatles rocked America? Or did you post it with the intention of the song
She Loves You being a sort of meta-commentary on our enviable, and now
infamous, bromance?

On Fri, Oct 21, 2011 at 5:13 AM, Paul Davis <paul.joseph.davis@gmail.com>wrote:

> http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s
>
> On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <nslater@tumbolia.org> wrote:
> > +1 on all the stuff Paul said.
> >
> > On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rnewson@apache.org>
> wrote:
> >
> >> I'll also note that the bug that killed round 1 of 1.1.1 was not found
> >> by any test we currently have. All it would have taken is a test that
> >> did any map call followed by almost any other bit of javascript (and
> >> sm 1.7.0).
> >>
> >> On 20 October 2011 21:22, Paul Davis <paul.joseph.davis@gmail.com>
> wrote:
> >> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <
> randall.leeds@gmail.com>
> >> wrote:
> >> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <
> paul.joseph.davis@gmail.com
> >> >wrote:
> >> >>
> >> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <randall@apache.org>
> >> wrote:
> >> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
> >> >>> lee@projectmastermind.com>wrote:
> >> >>> >
> >> >>> >>
> >> >>> >> > For what it's worth, a CLI based test system is what
I was
> >> imagining
> >> >>> >> > as well. Take Futon out of the mix and test CouchDB.
> >> >>> >>
> >> >>> >> IMO, If CouchDB is intended to be a server that can be
accessed
> from
> >> >>> >> the browser directly, then there should continue to be
some kind
> of
> >> >>> >> browser-based test suite that would serve to confirm this
> >> capability.
> >> >>> >>
> >> >>> >>
> >> >>> >> I have been looking closely at the Futon tests in 1.1.0
for the
> last
> >> >>> >> several days, with the idea that I might begin to clean
them up a
> >> bit
> >> >>> >> as time permits.
> >> >>> >>
> >> >>> >> I have found that, while some of these test failures are
totally
> >> bogus,
> >> >>> >> *some* of them actually do stem from real issues -- minor
> >> >>> >> incompatibilities between CouchDB's http interface, and
the
> internal
> >> >>> >> mechanisms of modern browsers (XHR, caching, etc).
> >> >>> >>
> >> >>> >> These are problems that we're not going to catch with
a
> stateless,
> >> >>> >> cache-less http client running on the CLI.  (I can provide
> examples)
> >> >>> >>
> >> >>> >> These issues have the potential to cause real problems
for
> >> >>> >> developers of real browser-based apps "in the wild". 
That means,
> >> >>> >> there's valuable info to be gathered from the browser
tests, Iff
> we
> >> >>> >> can clean them up, and make them behave consistently;
so that
> >> >>> >> when they fail or succeed, we can actually trust the results.
> >> >>> >>
> >> >>> >>
> >> >>> >> After digging around a good bit, I can see no reason why
the
> >> existing
> >> >>> >> tests couldn't be cleaned up and made to work correctly
in all
> >> current
> >> >>> >> versions of major browsers.  I also see no reason why
the same
> tests
> >> >>> >> couldn't be used successfully from the CLI and `make check`
as
> well.
> >> >>> >>
> >> >>> >> I do see significant benefits to using the same javascript
test
> code
> >> in
> >> >>> >> all environments we test.
> >> >>> >>
> >> >>> >> -Lee
> >> >>> >> (irc: coltr)
> >> >>> >>
> >> >>> >
> >> >>> >  +1
> >> >>> > Verify Installation could grow into a suite of browser/futon
tests
> >> that
> >> >>> > verify that futon (and apps in general) work, including
> interactions
> >> with
> >> >>> > proxies and the like.
> >> >>>
> >> >>> Sure. Client tests that test the client are fine.
> >> >>>
> >> >>> > The test suite for developers should run cleanly from the
CLI as
> part
> >> of
> >> >>> > make check, but continue to be exposed in futon. We should
work to
> be
> >> >>> sure
> >> >>> > they function as well as possible, for the reasons you provide.
> >> >>> >
> >> >>>
> >> >>> Blargh no. Server tests should be testing the server. The entire
> point
> >> >>> of moving to the command line is so that we don't have to maintain
> the
> >> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
> >> >>> release thread) and the wildly varying reports of test output.
The
> >> >>> situation is just a waste of time for everyone involved.
> >> >>>
> >> >>> > I think the JS testing situation is a great place for people
to
> jump
> >> in
> >> >>> and
> >> >>> > help out, especially with the browser environment diversity.
> >> >>> >
> >> >>>
> >> >>> Sure, but I don't see what this has to do with browsers.
> >> >>>
> >> >>
> >> >> People who aren't into the internals can help to fix the suite to
> work
> >> in
> >> >> different browser environments. That's all I meant.
> >> >>
> >> >
> >> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
> >> > "If these people exist, why do I not see anything in JIRA?"
> >> >
> >> >> I suggested that the CLI tests be exposed in Futon because I think
> there
> >> are
> >> >> probably some JS heads in this community who wouldn't have too much
> >> trouble
> >> >> fixing a lot of the user agent related issues in the test suite. I
> >> didn't
> >> >> mean to suggest that it should continue to be part of the release
> >> procedure
> >> >> (it shouldn't) or that we should feel 100% obligated to make sure
> they
> >> pass
> >> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
> >> about
> >> >> how keeping such tests around can sometimes expose interesting
> problems
> >> we
> >> >> wouldn't otherwise see, possible outside the CouchDB codebase even,
> is
> >> >> worthwhile.
> >> >>
> >> >> -Randall
> >> >>
> >> >
> >> > We've had these tests for three years or more now. Perhaps I'm just
> >> > being dense today but I can't think of a single specific case where
> >> > testing things in the browser has lead to a bug report/fix that we
> >> > wouldn't have found with pure CLI tests.
> >> >
> >> > The only thing that I'm aware that the tests have done for us is
> >> > required us to exert a nontrivial amount of effort to keep them
> >> > running in multiple browser environments. I have no interest in
> >> > maintaing these as tests runnable in the browser. I want to create a
> >> > CLI test environment that promotes stable, repeatable, concise tests.
> >> > Running these in a browser is the antithesis to such an environment.
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message