couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Wenk <andyw...@apache.org>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:45:20 GMT
ok this is a short update of Jan's description for new folks starting with
fixing the tests (thank's in advance).

1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
2. cd into the repo and `git checkout js-tests-on-2.0-wip`
3. run `make clean if this is not the first time you build
4. `./configure -c --disable-docs --disable-fauxton && make`

Now there are two options:

* run the whole test suite

 `make javascript`

* run a single test

`./dev/run --with-admin-party-please`
`./test/javascript/run test-filename` (in another tab)

Cheers

Andy

P.S.: Not torpedoing Jan's great description in any way :-)

On 12 October 2015 at 13:33, Jan Lehnardt <jan@apache.org> wrote:

> Oops, hits end too early.
>
>
> Okay -n 1 explains this
> > On 12 Oct 2015, at 13:32, Jan Lehnardt <jan@apache.org> wrote:
> >
> >
> >> On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
> >>
> >> Started with
> >>
> >> python dev/run -n 1 --with-admin-party-please &
> >>
> >> and export COUCH_HOST='http://127.0.0.1:15984'
> >>
> >>
> >> On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
> >>
> >>>
> >>>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
> >>>>
> >>>>> - instead of fixed test db names like `test_suite_db` that are used
> in
> >>>> almost every test,
> >>>>> we now generate a random database name for each test. The JS tests
> >>>> expected db
> >>>>> deletion to be synchronous, and that’s no longer the case in 2.0.
> >>> Instead
> >>>> of
> >>>>> error-prone polling to wait for db deletion, we just use new db
for
> >>> every
> >>>> test.
> >>>>> An added benefit now is that (some) tests can run in parallel.
> >>>>
> >>>> Little confused by this, PouchDB uses consistent names in its test
> suite
> >>>> (we had random
> >>>> ones for a while and moved away), we very much depend on the database
> >>> being
> >>>> deleted
> >>>> on the request returning and dont seem to have any problems with it
> >>> against
> >>>> master.
> >>>
> >>> Does the PouchDB test suite runs against the 5986 port, or with an n=1
> >>> cluster?
> >>> This is with the default n=3 cluster against 5986.
> >>>
> >>> Best
> >>> Jan
> >>> --
> >>>
> >>>>
> >>>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
> >>>>
> >>>>>
> >>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
> >>>>> sebastianrothbucher@googlemail.com> wrote:
> >>>>>>
> >>>>>> Hi Jan,
> >>>>>>
> >>>>>> I transferred some of my findings to your branch - pls. find
a PR
> for
> >>> the
> >>>>>> first few tests submitted 2 your branch. I put in TODOs where
either
> >>>>> there
> >>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853)
or we
> have
> >>>>> to
> >>>>>> do some more work (like making config available).
> >>>>>
> >>>>> Excellent, thanks! :)
> >>>>>
> >>>>>
> >>>>>> Will from the Pouch team had the great idea of using -n 1 to
> mitigate
> >>>>>> Heisenbugs due to distribution. I adopted it and it did work
quite
> >>> well.
> >>>>> It
> >>>>>> might make sense to put this into the dev/run call as a general
> rule,
> >>>>> esp.
> >>>>>> when checking for 201 (VS 202) status codes. Probably you know
more
> on
> >>>>>> that.
> >>>>>
> >>>>> We should carefully evaluate this :)
> >>>>>
> >>>>>
> >>>>>> Anyways: I think we have to keep using _config on the local
port as
> >>> many
> >>>>>> tests (like auth tests) depend on changed configurations. BTW:
w/
> -n 1
> >>>>> it's
> >>>>>> only one call, but calling 3 _config(s) migt also work.
> >>>>>> Hopefully (I just arrived at some test w/ that recently) we
can
> spare
> >>>>>> ourselves the _compact thing. I think we'd have to compact every
> single
> >>>>>> shard via local port, right? for the first tests, we can go
w/out
> as it
> >>>>>> seems. Maybe we should stick to that for a while.
> >>>>>
> >>>>> My concern with this is that as far as I care, the local port is
> >>> irrelevant
> >>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but
the
> >>> cluster
> >>>>> port is our publicised API for 2.0 and it’s the one that replaces
the
> >>> 1.x
> >>>>> API for people upgrading. So having any tests that run on the local
> port
> >>>>> of 2.0 don’t help us in any way (that’s not to say it still
useful to
> >>> have
> >>>>> tests against them for the moment).
> >>>>>
> >>>>> Many of the _config-dependent tests substitute out the _users or
> >>>>> _replicator
> >>>>> databases, maybe we can find a workaround for those by just using
the
> >>> stock
> >>>>> databases.
> >>>>>
> >>>>> For the other tests, maybe we split them out into their own “test
> >>> suites”
> >>>>> that start their own test cluster in the configuration they need
and
> >>> shut
> >>>>> it down afterwards.
> >>>>>
> >>>>> Best
> >>>>> Jan
> >>>>> --
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Looking fw 2 your thoughts
> >>>>>>
> >>>>>> Best
> >>>>>> Sebastian
> >>>>>>
> >>>>>> P.S::  Maybe we can adopt some stuff from
> >>>>>>
> >>>>>
> >>>
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
> >>>>>>
> >>>>>> P.P.S.: Will keep working on it, just not in the next few days,
> >>>>>> unfortunately ;-(
> >>>>>>
> >>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
> wrote:
> >>>>>>
> >>>>>>> (I need your help deciding what to do with a bunch of tests,
jump
> to
> >>>>>>> “Discussion” below if you don’t care about the details
before.)
> >>>>>>>
> >>>>>>> * * *
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I spent some more time getting the 1.x JavaScript tests
running on
> the
> >>>>>>> cluster port of 2.0. WIP here:
> >>>>> https://github.com/apache/couchdb/pull/354
> >>>>>>> — I’ve gotten a lot further, but there is still a lot
to do and
> >>> everyone
> >>>>>>> here can help, I’ve posted instructions on how to do so
further
> down.
> >>> No
> >>>>>>> Erlang knowledge needed, this is ideal for folks who want
to start
> >>> with
> >>>>>>> contributing to CouchDB.
> >>>>>>>
> >>>>>>>
> >>>>>>> Notable changes:
> >>>>>>>
> >>>>>>> - instead of fixed test db names like `test_suite_db` that
are
> used in
> >>>>>>> almost every test, we now generate a random database name
for each
> >>> test.
> >>>>>>> The JS tests expected db deletion to be synchronous, and
that’s no
> >>>>> longer
> >>>>>>> the case in 2.0. Instead of error-prone polling to wait
for db
> >>>>> deletion, we
> >>>>>>> just use new db for every test. An added benefit now is
that (some)
> >>>>> tests
> >>>>>>> can run in parallel.
> >>>>>>>
> >>>>>>> - tests now clean up after themselves (incomplete): tests
> previously
> >>>>> only
> >>>>>>> collectively used three or four databases, and deleted and
> re-created
> >>>>> them
> >>>>>>> before each test was run. Now with the change of creating
a new
> >>> database
> >>>>>>> per test, we also need to clean up databases at the end
of a test,
> >>>>> because
> >>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB
runs
> >>> out
> >>>>> of
> >>>>>>> file descriptors during the test run. The cleanup isn’t
100% done
> yet,
> >>>>> so
> >>>>>>> we should check for stray databases after JS test runs until
we’ve
> got
> >>>>> it
> >>>>>>> cleaned. One particular aspect here is when a test fails,
it never
> >>>>> reaches
> >>>>>>> the cleanup code, so we might have to wrap everything into
a
> try/catch
> >>>>>>> block.
> >>>>>>>
> >>>>>>> - I disabled all tests that still fail and made them so
they print
> the
> >>>>>>> reason why they are failing, so that fixing them should
be a lot
> >>> easier
> >>>>>>> now. Some of them print just “TODO”, their error cause
is yet to be
> >>>>>>> determined.
> >>>>>>>
> >>>>>>> - couchdb.query() which previously executed temporary views,
now
> under
> >>>>> the
> >>>>>>> hood creates a design doc with a random name and queries
that view.
> >>> This
> >>>>>>> fixes all tests relying on temp views without having to
change them
> >>> all
> >>>>>>> individually. This is because 2.0 doesn’t have temp views
anymore
> >>>>> (which is
> >>>>>>> a good thing).
> >>>>>>>
> >>>>>>> - restartServer() is no longer used (mostly).
> >>>>>>>
> >>>>>>> I already filed a few bugs that showed things missing in
the
> cluster
> >>> API
> >>>>>>> that we’d expect to be there:
> >>>>>>>
> >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
> >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
> >>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
> >>>>>>>
> >>>>>>> Based on this I am ever more convinced we should get this
test
> suite
> >>>>> work
> >>>>>>> against the 2.0 clustered API, because we’ll be finding
some more
> >>>>>>> inconsistencies that we should fix before we invite beta
testers.
> >>>>>>>
> >>>>>>>
> >>>>>>> ## Discussion:
> >>>>>>>
> >>>>>>> We have a whole number of tests that use the test-internal
function
> >>>>>>> run_on_modified_server. They use the _config API to test
CouchDB in
> >>>>>>> different configurations from the default one. All of these
tests
> now
> >>>>> throw
> >>>>>>> an error that _config isn’t available on the clustered
API. How
> should
> >>>>> we
> >>>>>>> handle them?
> >>>>>>>
> >>>>>>> The tests are:
> >>>>>>>
> >>>>>>> - erlang_views.js
> >>>>>>> - jsonp.js
> >>>>>>> - oauth_users_db.js
> >>>>>>> - oauth_users_db.js
> >>>>>>> - proxyauth.js
> >>>>>>> - reader_acl.js
> >>>>>>> - rewrite.js
> >>>>>>> - security_validation.js
> >>>>>>> - show_documents.js
> >>>>>>> - users_db.js
> >>>>>>> - view_errors.js
> >>>>>>>
> >>>>>>>
> >>>>>>> And a few tests aside from the proper compaction tests call
> _compact
> >>>>> which
> >>>>>>> is also not available on the cluster. What to do with them?
> >>>>>>>
> >>>>>>> - recreate_doc.js
> >>>>>>> - and a few more that are currently disabled for other reasons,
> sorry
> >>> I
> >>>>>>> don’t have a full list at this point.
> >>>>>>>
> >>>>>>> * * *
> >>>>>>>
> >>>>>>>
> >>>>>>> Bottom line: It’d be great if other folks here could join
in the
> fun.
> >>> No
> >>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP
API-work
> :)
> >>>>>>>
> >>>>>>>
> >>>>>>> ## How to help:
> >>>>>>>
> >>>>>>> 1. Check out
> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
> >>>>>>> 2. run `make clean && ./configure -c --disable-docs
> --disable-fauxton
> >>> &&
> >>>>>>> make`
> >>>>>>> 3. `make javascript` runs the whole test suite
> >>>>>>>
> >>>>>>> To run the whole test suite without waiting for rebar to
confirm
> that
> >>> no
> >>>>>>> Erlang code needs to be compiled, run `./test/javascript/run`
for
> the
> >>>>> whole
> >>>>>>> test suite, or `./test/javascript/run all_docs` to run an
> individual
> >>>>> test
> >>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other
file names
> >>> match
> >>>>>>> accordingly).
> >>>>>>>
> >>>>>>> Feel free to send me PRs against the branch on
> >>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip,
I’ll
> merge
> >>>>>>> them, so the PR on apache/couchdb gets updated.
> >>>>>>>
> >>>>>>> I’m looking forward to your contributions! :)
> >>>>>>>
> >>>>>>> If you have any questions or have trouble getting started
or just
> >>> would
> >>>>>>> like some hand-holding, let me know!
> >>>>>>>
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Jan
> >>>>>>> --
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> Professional Support for Apache CouchDB:
> >>>>> http://www.neighbourhood.ie/couchdb-support/
> >>>>>
> >>>>>
> >>>
> >>> --
> >>> Professional Support for Apache CouchDB:
> >>> http://www.neighbourhood.ie/couchdb-support/
> >>>
> >>>
> >
> > --
> > Professional Support for Apache CouchDB:
> > http://www.neighbourhood.ie/couchdb-support/
> >
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>


-- 
Andy Wenk
Hamburg - Germany
RockIt!

GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588

https://people.apache.org/keys/committer/andywenk.asc

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message