couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dale Harvey <d...@arandomurl.com>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:15:45 GMT
> - instead of fixed test db names like `test_suite_db` that are used in
almost every test,
> we now generate a random database name for each test. The JS tests
expected db
> deletion to be synchronous, and that’s no longer the case in 2.0. Instead
of
> error-prone polling to wait for db deletion, we just use new db for every
test.
> An added benefit now is that (some) tests can run in parallel.

Little confused by this, PouchDB uses consistent names in its test suite
(we had random
ones for a while and moved away), we very much depend on the database being
deleted
on the request returning and dont seem to have any problems with it against
master.

On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:

>
> > On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
> sebastianrothbucher@googlemail.com> wrote:
> >
> > Hi Jan,
> >
> > I transferred some of my findings to your branch - pls. find a PR for the
> > first few tests submitted 2 your branch. I put in TODOs where either
> there
> > is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have
> to
> > do some more work (like making config available).
>
> Excellent, thanks! :)
>
>
> > Will from the Pouch team had the great idea of using -n 1 to mitigate
> > Heisenbugs due to distribution. I adopted it and it did work quite well.
> It
> > might make sense to put this into the dev/run call as a general rule,
> esp.
> > when checking for 201 (VS 202) status codes. Probably you know more on
> > that.
>
> We should carefully evaluate this :)
>
>
> > Anyways: I think we have to keep using _config on the local port as many
> > tests (like auth tests) depend on changed configurations. BTW: w/ -n 1
> it's
> > only one call, but calling 3 _config(s) migt also work.
> > Hopefully (I just arrived at some test w/ that recently) we can spare
> > ourselves the _compact thing. I think we'd have to compact every single
> > shard via local port, right? for the first tests, we can go w/out as it
> > seems. Maybe we should stick to that for a while.
>
> My concern with this is that as far as I care, the local port is irrelevant
> to how CouchDB works. Of course it isn’t 100% irrelevant, but the cluster
> port is our publicised API for 2.0 and it’s the one that replaces the 1.x
> API for people upgrading. So having any tests that run on the local port
> of 2.0 don’t help us in any way (that’s not to say it still useful to have
> tests against them for the moment).
>
> Many of the _config-dependent tests substitute out the _users or
> _replicator
> databases, maybe we can find a workaround for those by just using the stock
> databases.
>
> For the other tests, maybe we split them out into their own “test suites”
> that start their own test cluster in the configuration they need and shut
> it down afterwards.
>
> Best
> Jan
> --
>
>
> >
> > Looking fw 2 your thoughts
> >
> > Best
> >    Sebastian
> >
> > P.S::  Maybe we can adopt some stuff from
> >
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
> >
> > P.P.S.: Will keep working on it, just not in the next few days,
> > unfortunately ;-(
> >
> > On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org> wrote:
> >
> >> (I need your help deciding what to do with a bunch of tests, jump to
> >> “Discussion” below if you don’t care about the details before.)
> >>
> >> * * *
> >>
> >> Hi all,
> >>
> >> I spent some more time getting the 1.x JavaScript tests running on the
> >> cluster port of 2.0. WIP here:
> https://github.com/apache/couchdb/pull/354
> >> — I’ve gotten a lot further, but there is still a lot to do and everyone
> >> here can help, I’ve posted instructions on how to do so further down. No
> >> Erlang knowledge needed, this is ideal for folks who want to start with
> >> contributing to CouchDB.
> >>
> >>
> >> Notable changes:
> >>
> >> - instead of fixed test db names like `test_suite_db` that are used in
> >> almost every test, we now generate a random database name for each test.
> >> The JS tests expected db deletion to be synchronous, and that’s no
> longer
> >> the case in 2.0. Instead of error-prone polling to wait for db
> deletion, we
> >> just use new db for every test. An added benefit now is that (some)
> tests
> >> can run in parallel.
> >>
> >> - tests now clean up after themselves (incomplete): tests previously
> only
> >> collectively used three or four databases, and deleted and re-created
> them
> >> before each test was run. Now with the change of creating a new database
> >> per test, we also need to clean up databases at the end of a test,
> because
> >> otherwise we a) leave a lot of databases around and b) CouchDB runs out
> of
> >> file descriptors during the test run. The cleanup isn’t 100% done yet,
> so
> >> we should check for stray databases after JS test runs until we’ve got
> it
> >> cleaned. One particular aspect here is when a test fails, it never
> reaches
> >> the cleanup code, so we might have to wrap everything into a try/catch
> >> block.
> >>
> >> - I disabled all tests that still fail and made them so they print the
> >> reason why they are failing, so that fixing them should be a lot easier
> >> now. Some of them print just “TODO”, their error cause is yet to be
> >> determined.
> >>
> >> - couchdb.query() which previously executed temporary views, now under
> the
> >> hood creates a design doc with a random name and queries that view. This
> >> fixes all tests relying on temp views without having to change them all
> >> individually. This is because 2.0 doesn’t have temp views anymore
> (which is
> >> a good thing).
> >>
> >> - restartServer() is no longer used (mostly).
> >>
> >> I already filed a few bugs that showed things missing in the cluster API
> >> that we’d expect to be there:
> >>
> >> - https://issues.apache.org/jira/browse/COUCHDB-2849
> >> - https://issues.apache.org/jira/browse/COUCHDB-2850
> >> - https://issues.apache.org/jira/browse/COUCHDB-2851
> >>
> >> Based on this I am ever more convinced we should get this test suite
> work
> >> against the 2.0 clustered API, because we’ll be finding some more
> >> inconsistencies that we should fix before we invite beta testers.
> >>
> >>
> >> ## Discussion:
> >>
> >> We have a whole number of tests that use the test-internal function
> >> run_on_modified_server. They use the _config API to test CouchDB in
> >> different configurations from the default one. All of these tests now
> throw
> >> an error that _config isn’t available on the clustered API. How should
> we
> >> handle them?
> >>
> >> The tests are:
> >>
> >> - erlang_views.js
> >> - jsonp.js
> >> - oauth_users_db.js
> >> - oauth_users_db.js
> >> - proxyauth.js
> >> - reader_acl.js
> >> - rewrite.js
> >> - security_validation.js
> >> - show_documents.js
> >> - users_db.js
> >> - view_errors.js
> >>
> >>
> >> And a few tests aside from the proper compaction tests call _compact
> which
> >> is also not available on the cluster. What to do with them?
> >>
> >> - recreate_doc.js
> >> - and a few more that are currently disabled for other reasons, sorry I
> >> don’t have a full list at this point.
> >>
> >> * * *
> >>
> >>
> >> Bottom line: It’d be great if other folks here could join in the fun. No
> >> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :)
> >>
> >>
> >> ## How to help:
> >>
> >> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
> >> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton
&&
> >> make`
> >> 3. `make javascript` runs the whole test suite
> >>
> >> To run the whole test suite without waiting for rebar to confirm that no
> >> Erlang code needs to be compiled, run `./test/javascript/run` for the
> whole
> >> test suite, or `./test/javascript/run all_docs` to run an individual
> test
> >> (all_docs runs test/javascript/tests/all_docs.js, other file names match
> >> accordingly).
> >>
> >> Feel free to send me PRs against the branch on
> >> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge
> >> them, so the PR on apache/couchdb gets updated.
> >>
> >> I’m looking forward to your contributions! :)
> >>
> >> If you have any questions or have trouble getting started or just would
> >> like some hand-holding, let me know!
> >>
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>
> >>
> >>
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message