couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Rothbucher <sebastianrothbuc...@googlemail.com>
Subject Re: The JavaScript Test Suite
Date Sat, 17 Oct 2015 14:45:21 GMT
Hi all,

got a little further at https://github.com/janl/couchdb/pull/3/files

This is it so far:

# for i in test/javascript/tests/a* test/javascript/tests/b*
test/javascript/tests/change*; dev/run -n 1 -q --with-admin-party-please
test/javascript/run $i; done

test/javascript/tests/all_docs.js   pass

test/javascript/tests/attachment_names.js   pass

test/javascript/tests/attachment_paths.js   pass

test/javascript/tests/attachment_ranges.js   pass

test/javascript/tests/attachments.js   pass

test/javascript/tests/attachments_multipart.js   pass

test/javascript/tests/attachment_views.js   pass

test/javascript/tests/auth_cache.js   TODO  pass

test/javascript/tests/basics.js   pass

test/javascript/tests/batch_save.js   pass

test/javascript/tests/bulk_docs.js   pass

test/javascript/tests/changes.js   pass


It uncovers some problems with latest=... in the URL and Last event ID a
header param that might be worth a ticket


We're rolling... and still some work 2 do ;-)


    Sebastian


On Wed, Oct 14, 2015 at 12:23 PM, Jan Lehnardt <jan@apache.org> wrote:

> I’ve run the 1.x test suite against an -n1 cluster and it doesn’t improve
> things
> significantly, although I haven’t looked at database deletes, as I’ve got
> these
> covered for now.
>
> So there is still some work that needs doing :)
>
> Best
> Jan
> --
>
> > On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
> >
> > Started with
> >
> > python dev/run -n 1 --with-admin-party-please &
> >
> > and export COUCH_HOST='http://127.0.0.1:15984'
> >
> >
> > On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
> >
> >>
> >>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
> >>>
> >>>> - instead of fixed test db names like `test_suite_db` that are used
in
> >>> almost every test,
> >>>> we now generate a random database name for each test. The JS tests
> >>> expected db
> >>>> deletion to be synchronous, and that’s no longer the case in 2.0.
> >> Instead
> >>> of
> >>>> error-prone polling to wait for db deletion, we just use new db for
> >> every
> >>> test.
> >>>> An added benefit now is that (some) tests can run in parallel.
> >>>
> >>> Little confused by this, PouchDB uses consistent names in its test
> suite
> >>> (we had random
> >>> ones for a while and moved away), we very much depend on the database
> >> being
> >>> deleted
> >>> on the request returning and dont seem to have any problems with it
> >> against
> >>> master.
> >>
> >> Does the PouchDB test suite runs against the 5986 port, or with an n=1
> >> cluster?
> >> This is with the default n=3 cluster against 5986.
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>>
> >>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
> >>>
> >>>>
> >>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
> >>>> sebastianrothbucher@googlemail.com> wrote:
> >>>>>
> >>>>> Hi Jan,
> >>>>>
> >>>>> I transferred some of my findings to your branch - pls. find a PR
for
> >> the
> >>>>> first few tests submitted 2 your branch. I put in TODOs where either
> >>>> there
> >>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or
we
> have
> >>>> to
> >>>>> do some more work (like making config available).
> >>>>
> >>>> Excellent, thanks! :)
> >>>>
> >>>>
> >>>>> Will from the Pouch team had the great idea of using -n 1 to mitigate
> >>>>> Heisenbugs due to distribution. I adopted it and it did work quite
> >> well.
> >>>> It
> >>>>> might make sense to put this into the dev/run call as a general
rule,
> >>>> esp.
> >>>>> when checking for 201 (VS 202) status codes. Probably you know more
> on
> >>>>> that.
> >>>>
> >>>> We should carefully evaluate this :)
> >>>>
> >>>>
> >>>>> Anyways: I think we have to keep using _config on the local port
as
> >> many
> >>>>> tests (like auth tests) depend on changed configurations. BTW: w/
-n
> 1
> >>>> it's
> >>>>> only one call, but calling 3 _config(s) migt also work.
> >>>>> Hopefully (I just arrived at some test w/ that recently) we can
spare
> >>>>> ourselves the _compact thing. I think we'd have to compact every
> single
> >>>>> shard via local port, right? for the first tests, we can go w/out
as
> it
> >>>>> seems. Maybe we should stick to that for a while.
> >>>>
> >>>> My concern with this is that as far as I care, the local port is
> >> irrelevant
> >>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the
> >> cluster
> >>>> port is our publicised API for 2.0 and it’s the one that replaces
the
> >> 1.x
> >>>> API for people upgrading. So having any tests that run on the local
> port
> >>>> of 2.0 don’t help us in any way (that’s not to say it still useful
to
> >> have
> >>>> tests against them for the moment).
> >>>>
> >>>> Many of the _config-dependent tests substitute out the _users or
> >>>> _replicator
> >>>> databases, maybe we can find a workaround for those by just using the
> >> stock
> >>>> databases.
> >>>>
> >>>> For the other tests, maybe we split them out into their own “test
> >> suites”
> >>>> that start their own test cluster in the configuration they need and
> >> shut
> >>>> it down afterwards.
> >>>>
> >>>> Best
> >>>> Jan
> >>>> --
> >>>>
> >>>>
> >>>>>
> >>>>> Looking fw 2 your thoughts
> >>>>>
> >>>>> Best
> >>>>>  Sebastian
> >>>>>
> >>>>> P.S::  Maybe we can adopt some stuff from
> >>>>>
> >>>>
> >>
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
> >>>>>
> >>>>> P.P.S.: Will keep working on it, just not in the next few days,
> >>>>> unfortunately ;-(
> >>>>>
> >>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
> wrote:
> >>>>>
> >>>>>> (I need your help deciding what to do with a bunch of tests,
jump to
> >>>>>> “Discussion” below if you don’t care about the details
before.)
> >>>>>>
> >>>>>> * * *
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I spent some more time getting the 1.x JavaScript tests running
on
> the
> >>>>>> cluster port of 2.0. WIP here:
> >>>> https://github.com/apache/couchdb/pull/354
> >>>>>> — I’ve gotten a lot further, but there is still a lot to
do and
> >> everyone
> >>>>>> here can help, I’ve posted instructions on how to do so further
> down.
> >> No
> >>>>>> Erlang knowledge needed, this is ideal for folks who want to
start
> >> with
> >>>>>> contributing to CouchDB.
> >>>>>>
> >>>>>>
> >>>>>> Notable changes:
> >>>>>>
> >>>>>> - instead of fixed test db names like `test_suite_db` that are
used
> in
> >>>>>> almost every test, we now generate a random database name for
each
> >> test.
> >>>>>> The JS tests expected db deletion to be synchronous, and that’s
no
> >>>> longer
> >>>>>> the case in 2.0. Instead of error-prone polling to wait for
db
> >>>> deletion, we
> >>>>>> just use new db for every test. An added benefit now is that
(some)
> >>>> tests
> >>>>>> can run in parallel.
> >>>>>>
> >>>>>> - tests now clean up after themselves (incomplete): tests previously
> >>>> only
> >>>>>> collectively used three or four databases, and deleted and
> re-created
> >>>> them
> >>>>>> before each test was run. Now with the change of creating a
new
> >> database
> >>>>>> per test, we also need to clean up databases at the end of a
test,
> >>>> because
> >>>>>> otherwise we a) leave a lot of databases around and b) CouchDB
runs
> >> out
> >>>> of
> >>>>>> file descriptors during the test run. The cleanup isn’t 100%
done
> yet,
> >>>> so
> >>>>>> we should check for stray databases after JS test runs until
we’ve
> got
> >>>> it
> >>>>>> cleaned. One particular aspect here is when a test fails, it
never
> >>>> reaches
> >>>>>> the cleanup code, so we might have to wrap everything into a
> try/catch
> >>>>>> block.
> >>>>>>
> >>>>>> - I disabled all tests that still fail and made them so they
print
> the
> >>>>>> reason why they are failing, so that fixing them should be a
lot
> >> easier
> >>>>>> now. Some of them print just “TODO”, their error cause is
yet to be
> >>>>>> determined.
> >>>>>>
> >>>>>> - couchdb.query() which previously executed temporary views,
now
> under
> >>>> the
> >>>>>> hood creates a design doc with a random name and queries that
view.
> >> This
> >>>>>> fixes all tests relying on temp views without having to change
them
> >> all
> >>>>>> individually. This is because 2.0 doesn’t have temp views
anymore
> >>>> (which is
> >>>>>> a good thing).
> >>>>>>
> >>>>>> - restartServer() is no longer used (mostly).
> >>>>>>
> >>>>>> I already filed a few bugs that showed things missing in the
cluster
> >> API
> >>>>>> that we’d expect to be there:
> >>>>>>
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
> >>>>>>
> >>>>>> Based on this I am ever more convinced we should get this test
suite
> >>>> work
> >>>>>> against the 2.0 clustered API, because we’ll be finding some
more
> >>>>>> inconsistencies that we should fix before we invite beta testers.
> >>>>>>
> >>>>>>
> >>>>>> ## Discussion:
> >>>>>>
> >>>>>> We have a whole number of tests that use the test-internal function
> >>>>>> run_on_modified_server. They use the _config API to test CouchDB
in
> >>>>>> different configurations from the default one. All of these
tests
> now
> >>>> throw
> >>>>>> an error that _config isn’t available on the clustered API.
How
> should
> >>>> we
> >>>>>> handle them?
> >>>>>>
> >>>>>> The tests are:
> >>>>>>
> >>>>>> - erlang_views.js
> >>>>>> - jsonp.js
> >>>>>> - oauth_users_db.js
> >>>>>> - oauth_users_db.js
> >>>>>> - proxyauth.js
> >>>>>> - reader_acl.js
> >>>>>> - rewrite.js
> >>>>>> - security_validation.js
> >>>>>> - show_documents.js
> >>>>>> - users_db.js
> >>>>>> - view_errors.js
> >>>>>>
> >>>>>>
> >>>>>> And a few tests aside from the proper compaction tests call
_compact
> >>>> which
> >>>>>> is also not available on the cluster. What to do with them?
> >>>>>>
> >>>>>> - recreate_doc.js
> >>>>>> - and a few more that are currently disabled for other reasons,
> sorry
> >> I
> >>>>>> don’t have a full list at this point.
> >>>>>>
> >>>>>> * * *
> >>>>>>
> >>>>>>
> >>>>>> Bottom line: It’d be great if other folks here could join
in the
> fun.
> >> No
> >>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work
> :)
> >>>>>>
> >>>>>>
> >>>>>> ## How to help:
> >>>>>>
> >>>>>> 1. Check out
> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
> >>>>>> 2. run `make clean && ./configure -c --disable-docs
> --disable-fauxton
> >> &&
> >>>>>> make`
> >>>>>> 3. `make javascript` runs the whole test suite
> >>>>>>
> >>>>>> To run the whole test suite without waiting for rebar to confirm
> that
> >> no
> >>>>>> Erlang code needs to be compiled, run `./test/javascript/run`
for
> the
> >>>> whole
> >>>>>> test suite, or `./test/javascript/run all_docs` to run an individual
> >>>> test
> >>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file
names
> >> match
> >>>>>> accordingly).
> >>>>>>
> >>>>>> Feel free to send me PRs against the branch on
> >>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll
> merge
> >>>>>> them, so the PR on apache/couchdb gets updated.
> >>>>>>
> >>>>>> I’m looking forward to your contributions! :)
> >>>>>>
> >>>>>> If you have any questions or have trouble getting started or
just
> >> would
> >>>>>> like some hand-holding, let me know!
> >>>>>>
> >>>>>>
> >>>>>> Best
> >>>>>> Jan
> >>>>>> --
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>> --
> >>>> Professional Support for Apache CouchDB:
> >>>> http://www.neighbourhood.ie/couchdb-support/
> >>>>
> >>>>
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> http://www.neighbourhood.ie/couchdb-support/
> >>
> >>
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message