couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dale Harvey <d...@arandomurl.com>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:30:53 GMT
Started with

python dev/run -n 1 --with-admin-party-please &

and export COUCH_HOST='http://127.0.0.1:15984'


On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:

>
> > On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
> >
> >> - instead of fixed test db names like `test_suite_db` that are used in
> > almost every test,
> >> we now generate a random database name for each test. The JS tests
> > expected db
> >> deletion to be synchronous, and that’s no longer the case in 2.0.
> Instead
> > of
> >> error-prone polling to wait for db deletion, we just use new db for
> every
> > test.
> >> An added benefit now is that (some) tests can run in parallel.
> >
> > Little confused by this, PouchDB uses consistent names in its test suite
> > (we had random
> > ones for a while and moved away), we very much depend on the database
> being
> > deleted
> > on the request returning and dont seem to have any problems with it
> against
> > master.
>
> Does the PouchDB test suite runs against the 5986 port, or with an n=1
> cluster?
> This is with the default n=3 cluster against 5986.
>
> Best
> Jan
> --
>
> >
> > On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
> >
> >>
> >>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
> >> sebastianrothbucher@googlemail.com> wrote:
> >>>
> >>> Hi Jan,
> >>>
> >>> I transferred some of my findings to your branch - pls. find a PR for
> the
> >>> first few tests submitted 2 your branch. I put in TODOs where either
> >> there
> >>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have
> >> to
> >>> do some more work (like making config available).
> >>
> >> Excellent, thanks! :)
> >>
> >>
> >>> Will from the Pouch team had the great idea of using -n 1 to mitigate
> >>> Heisenbugs due to distribution. I adopted it and it did work quite
> well.
> >> It
> >>> might make sense to put this into the dev/run call as a general rule,
> >> esp.
> >>> when checking for 201 (VS 202) status codes. Probably you know more on
> >>> that.
> >>
> >> We should carefully evaluate this :)
> >>
> >>
> >>> Anyways: I think we have to keep using _config on the local port as
> many
> >>> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1
> >> it's
> >>> only one call, but calling 3 _config(s) migt also work.
> >>> Hopefully (I just arrived at some test w/ that recently) we can spare
> >>> ourselves the _compact thing. I think we'd have to compact every single
> >>> shard via local port, right? for the first tests, we can go w/out as it
> >>> seems. Maybe we should stick to that for a while.
> >>
> >> My concern with this is that as far as I care, the local port is
> irrelevant
> >> to how CouchDB works. Of course it isn’t 100% irrelevant, but the
> cluster
> >> port is our publicised API for 2.0 and it’s the one that replaces the
> 1.x
> >> API for people upgrading. So having any tests that run on the local port
> >> of 2.0 don’t help us in any way (that’s not to say it still useful to
> have
> >> tests against them for the moment).
> >>
> >> Many of the _config-dependent tests substitute out the _users or
> >> _replicator
> >> databases, maybe we can find a workaround for those by just using the
> stock
> >> databases.
> >>
> >> For the other tests, maybe we split them out into their own “test
> suites”
> >> that start their own test cluster in the configuration they need and
> shut
> >> it down afterwards.
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>
> >>>
> >>> Looking fw 2 your thoughts
> >>>
> >>> Best
> >>>   Sebastian
> >>>
> >>> P.S::  Maybe we can adopt some stuff from
> >>>
> >>
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
> >>>
> >>> P.P.S.: Will keep working on it, just not in the next few days,
> >>> unfortunately ;-(
> >>>
> >>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org> wrote:
> >>>
> >>>> (I need your help deciding what to do with a bunch of tests, jump to
> >>>> “Discussion” below if you don’t care about the details before.)
> >>>>
> >>>> * * *
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I spent some more time getting the 1.x JavaScript tests running on the
> >>>> cluster port of 2.0. WIP here:
> >> https://github.com/apache/couchdb/pull/354
> >>>> — I’ve gotten a lot further, but there is still a lot to do and
> everyone
> >>>> here can help, I’ve posted instructions on how to do so further down.
> No
> >>>> Erlang knowledge needed, this is ideal for folks who want to start
> with
> >>>> contributing to CouchDB.
> >>>>
> >>>>
> >>>> Notable changes:
> >>>>
> >>>> - instead of fixed test db names like `test_suite_db` that are used
in
> >>>> almost every test, we now generate a random database name for each
> test.
> >>>> The JS tests expected db deletion to be synchronous, and that’s no
> >> longer
> >>>> the case in 2.0. Instead of error-prone polling to wait for db
> >> deletion, we
> >>>> just use new db for every test. An added benefit now is that (some)
> >> tests
> >>>> can run in parallel.
> >>>>
> >>>> - tests now clean up after themselves (incomplete): tests previously
> >> only
> >>>> collectively used three or four databases, and deleted and re-created
> >> them
> >>>> before each test was run. Now with the change of creating a new
> database
> >>>> per test, we also need to clean up databases at the end of a test,
> >> because
> >>>> otherwise we a) leave a lot of databases around and b) CouchDB runs
> out
> >> of
> >>>> file descriptors during the test run. The cleanup isn’t 100% done
yet,
> >> so
> >>>> we should check for stray databases after JS test runs until we’ve
got
> >> it
> >>>> cleaned. One particular aspect here is when a test fails, it never
> >> reaches
> >>>> the cleanup code, so we might have to wrap everything into a try/catch
> >>>> block.
> >>>>
> >>>> - I disabled all tests that still fail and made them so they print the
> >>>> reason why they are failing, so that fixing them should be a lot
> easier
> >>>> now. Some of them print just “TODO”, their error cause is yet to
be
> >>>> determined.
> >>>>
> >>>> - couchdb.query() which previously executed temporary views, now under
> >> the
> >>>> hood creates a design doc with a random name and queries that view.
> This
> >>>> fixes all tests relying on temp views without having to change them
> all
> >>>> individually. This is because 2.0 doesn’t have temp views anymore
> >> (which is
> >>>> a good thing).
> >>>>
> >>>> - restartServer() is no longer used (mostly).
> >>>>
> >>>> I already filed a few bugs that showed things missing in the cluster
> API
> >>>> that we’d expect to be there:
> >>>>
> >>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
> >>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
> >>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
> >>>>
> >>>> Based on this I am ever more convinced we should get this test suite
> >> work
> >>>> against the 2.0 clustered API, because we’ll be finding some more
> >>>> inconsistencies that we should fix before we invite beta testers.
> >>>>
> >>>>
> >>>> ## Discussion:
> >>>>
> >>>> We have a whole number of tests that use the test-internal function
> >>>> run_on_modified_server. They use the _config API to test CouchDB in
> >>>> different configurations from the default one. All of these tests now
> >> throw
> >>>> an error that _config isn’t available on the clustered API. How should
> >> we
> >>>> handle them?
> >>>>
> >>>> The tests are:
> >>>>
> >>>> - erlang_views.js
> >>>> - jsonp.js
> >>>> - oauth_users_db.js
> >>>> - oauth_users_db.js
> >>>> - proxyauth.js
> >>>> - reader_acl.js
> >>>> - rewrite.js
> >>>> - security_validation.js
> >>>> - show_documents.js
> >>>> - users_db.js
> >>>> - view_errors.js
> >>>>
> >>>>
> >>>> And a few tests aside from the proper compaction tests call _compact
> >> which
> >>>> is also not available on the cluster. What to do with them?
> >>>>
> >>>> - recreate_doc.js
> >>>> - and a few more that are currently disabled for other reasons, sorry
> I
> >>>> don’t have a full list at this point.
> >>>>
> >>>> * * *
> >>>>
> >>>>
> >>>> Bottom line: It’d be great if other folks here could join in the fun.
> No
> >>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :)
> >>>>
> >>>>
> >>>> ## How to help:
> >>>>
> >>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
> >>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton
> &&
> >>>> make`
> >>>> 3. `make javascript` runs the whole test suite
> >>>>
> >>>> To run the whole test suite without waiting for rebar to confirm that
> no
> >>>> Erlang code needs to be compiled, run `./test/javascript/run` for the
> >> whole
> >>>> test suite, or `./test/javascript/run all_docs` to run an individual
> >> test
> >>>> (all_docs runs test/javascript/tests/all_docs.js, other file names
> match
> >>>> accordingly).
> >>>>
> >>>> Feel free to send me PRs against the branch on
> >>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge
> >>>> them, so the PR on apache/couchdb gets updated.
> >>>>
> >>>> I’m looking forward to your contributions! :)
> >>>>
> >>>> If you have any questions or have trouble getting started or just
> would
> >>>> like some hand-holding, let me know!
> >>>>
> >>>>
> >>>> Best
> >>>> Jan
> >>>> --
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> http://www.neighbourhood.ie/couchdb-support/
> >>
> >>
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message