Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: <F04D27A6-2A37-4C4F-879C-CBBA499EC5FC@apache.org>
References: <6443EF40-E58C-4804-869D-3BDA1132C6E9@apache.org>
	<CA+pq5fqP6T_2M+rkAgVRV69NzVmor8DmozJTtrvkfMX4kd=5pw@mail.gmail.com>
	<95DEDD8F-34F2-4D3C-A037-CD787D765B00@apache.org>
	<CAD2UGCU+4kYbihbgY48qf37mLWA2JCsgG7tpkaXBe8yuYvEbyw@mail.gmail.com>
	<A7A9AAE3-5CB4-43A9-9809-3110EF1204E8@apache.org>
	<CAD2UGCXJMjRnWSM2JOkA9fKeMqhBUYyi+iHR4pn8F7NaMdk_aA@mail.gmail.com>
	<F04D27A6-2A37-4C4F-879C-CBBA499EC5FC@apache.org>
Date: Sat, 17 Oct 2015 16:45:21 +0200
Message-ID: 
 <CA+pq5fpqxR5GUfhjOx2ArS4f-1t9iBkmcSGbSFa4r+ueEy24Hw@mail.gmail.com>
Subject: Re: The JavaScript Test Suite
From: Sebastian Rothbucher <sebastianrothbucher@googlemail.com>
To: "dev@couchdb.apache.org" <dev@couchdb.apache.org>
Content-Type: multipart/alternative; boundary=001a11428e70304a0205224df6bb

--001a11428e70304a0205224df6bb
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi all,

got a little further at https://github.com/janl/couchdb/pull/3/files

This is it so far:

# for i in test/javascript/tests/a* test/javascript/tests/b*
test/javascript/tests/change*; dev/run -n 1 -q --with-admin-party-please
test/javascript/run $i; done

test/javascript/tests/all_docs.js   pass

test/javascript/tests/attachment_names.js   pass

test/javascript/tests/attachment_paths.js   pass

test/javascript/tests/attachment_ranges.js   pass

test/javascript/tests/attachments.js   pass

test/javascript/tests/attachments_multipart.js   pass

test/javascript/tests/attachment_views.js   pass

test/javascript/tests/auth_cache.js   TODO  pass

test/javascript/tests/basics.js   pass

test/javascript/tests/batch_save.js   pass

test/javascript/tests/bulk_docs.js   pass

test/javascript/tests/changes.js   pass


It uncovers some problems with latest=3D... in the URL and Last event ID a
header param that might be worth a ticket


We're rolling... and still some work 2 do ;-)


    Sebastian


On Wed, Oct 14, 2015 at 12:23 PM, Jan Lehnardt <jan@apache.org> wrote:

> I=E2=80=99ve run the 1.x test suite against an -n1 cluster and it doesn=
=E2=80=99t improve
> things
> significantly, although I haven=E2=80=99t looked at database deletes, as =
I=E2=80=99ve got
> these
> covered for now.
>
> So there is still some work that needs doing :)
>
> Best
> Jan
> --
>
> > On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
> >
> > Started with
> >
> > python dev/run -n 1 --with-admin-party-please &
> >
> > and export COUCH_HOST=3D'http://127.0.0.1:15984'
> >
> >
> > On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
> >
> >>
> >>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
> >>>
> >>>> - instead of fixed test db names like `test_suite_db` that are used =
in
> >>> almost every test,
> >>>> we now generate a random database name for each test. The JS tests
> >>> expected db
> >>>> deletion to be synchronous, and that=E2=80=99s no longer the case in=
 2.0.
> >> Instead
> >>> of
> >>>> error-prone polling to wait for db deletion, we just use new db for
> >> every
> >>> test.
> >>>> An added benefit now is that (some) tests can run in parallel.
> >>>
> >>> Little confused by this, PouchDB uses consistent names in its test
> suite
> >>> (we had random
> >>> ones for a while and moved away), we very much depend on the database
> >> being
> >>> deleted
> >>> on the request returning and dont seem to have any problems with it
> >> against
> >>> master.
> >>
> >> Does the PouchDB test suite runs against the 5986 port, or with an n=
=3D1
> >> cluster?
> >> This is with the default n=3D3 cluster against 5986.
> >>
> >> Best
> >> Jan
> >> --
> >>
> >>>
> >>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
> >>>
> >>>>
> >>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
> >>>> sebastianrothbucher@googlemail.com> wrote:
> >>>>>
> >>>>> Hi Jan,
> >>>>>
> >>>>> I transferred some of my findings to your branch - pls. find a PR f=
or
> >> the
> >>>>> first few tests submitted 2 your branch. I put in TODOs where eithe=
r
> >>>> there
> >>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we
> have
> >>>> to
> >>>>> do some more work (like making config available).
> >>>>
> >>>> Excellent, thanks! :)
> >>>>
> >>>>
> >>>>> Will from the Pouch team had the great idea of using -n 1 to mitiga=
te
> >>>>> Heisenbugs due to distribution. I adopted it and it did work quite
> >> well.
> >>>> It
> >>>>> might make sense to put this into the dev/run call as a general rul=
e,
> >>>> esp.
> >>>>> when checking for 201 (VS 202) status codes. Probably you know more
> on
> >>>>> that.
> >>>>
> >>>> We should carefully evaluate this :)
> >>>>
> >>>>
> >>>>> Anyways: I think we have to keep using _config on the local port as
> >> many
> >>>>> tests (like auth tests) depend on changed configurations. BTW: w/ -=
n
> 1
> >>>> it's
> >>>>> only one call, but calling 3 _config(s) migt also work.
> >>>>> Hopefully (I just arrived at some test w/ that recently) we can spa=
re
> >>>>> ourselves the _compact thing. I think we'd have to compact every
> single
> >>>>> shard via local port, right? for the first tests, we can go w/out a=
s
> it
> >>>>> seems. Maybe we should stick to that for a while.
> >>>>
> >>>> My concern with this is that as far as I care, the local port is
> >> irrelevant
> >>>> to how CouchDB works. Of course it isn=E2=80=99t 100% irrelevant, bu=
t the
> >> cluster
> >>>> port is our publicised API for 2.0 and it=E2=80=99s the one that rep=
laces the
> >> 1.x
> >>>> API for people upgrading. So having any tests that run on the local
> port
> >>>> of 2.0 don=E2=80=99t help us in any way (that=E2=80=99s not to say i=
t still useful to
> >> have
> >>>> tests against them for the moment).
> >>>>
> >>>> Many of the _config-dependent tests substitute out the _users or
> >>>> _replicator
> >>>> databases, maybe we can find a workaround for those by just using th=
e
> >> stock
> >>>> databases.
> >>>>
> >>>> For the other tests, maybe we split them out into their own =E2=80=
=9Ctest
> >> suites=E2=80=9D
> >>>> that start their own test cluster in the configuration they need and
> >> shut
> >>>> it down afterwards.
> >>>>
> >>>> Best
> >>>> Jan
> >>>> --
> >>>>
> >>>>
> >>>>>
> >>>>> Looking fw 2 your thoughts
> >>>>>
> >>>>> Best
> >>>>>  Sebastian
> >>>>>
> >>>>> P.S::  Maybe we can adopt some stuff from
> >>>>>
> >>>>
> >>
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:cl=
ustertest
> >>>>>
> >>>>> P.P.S.: Will keep working on it, just not in the next few days,
> >>>>> unfortunately ;-(
> >>>>>
> >>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
> wrote:
> >>>>>
> >>>>>> (I need your help deciding what to do with a bunch of tests, jump =
to
> >>>>>> =E2=80=9CDiscussion=E2=80=9D below if you don=E2=80=99t care about=
 the details before.)
> >>>>>>
> >>>>>> * * *
> >>>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I spent some more time getting the 1.x JavaScript tests running on
> the
> >>>>>> cluster port of 2.0. WIP here:
> >>>> https://github.com/apache/couchdb/pull/354
> >>>>>> =E2=80=94 I=E2=80=99ve gotten a lot further, but there is still a =
lot to do and
> >> everyone
> >>>>>> here can help, I=E2=80=99ve posted instructions on how to do so fu=
rther
> down.
> >> No
> >>>>>> Erlang knowledge needed, this is ideal for folks who want to start
> >> with
> >>>>>> contributing to CouchDB.
> >>>>>>
> >>>>>>
> >>>>>> Notable changes:
> >>>>>>
> >>>>>> - instead of fixed test db names like `test_suite_db` that are use=
d
> in
> >>>>>> almost every test, we now generate a random database name for each
> >> test.
> >>>>>> The JS tests expected db deletion to be synchronous, and that=E2=
=80=99s no
> >>>> longer
> >>>>>> the case in 2.0. Instead of error-prone polling to wait for db
> >>>> deletion, we
> >>>>>> just use new db for every test. An added benefit now is that (some=
)
> >>>> tests
> >>>>>> can run in parallel.
> >>>>>>
> >>>>>> - tests now clean up after themselves (incomplete): tests previous=
ly
> >>>> only
> >>>>>> collectively used three or four databases, and deleted and
> re-created
> >>>> them
> >>>>>> before each test was run. Now with the change of creating a new
> >> database
> >>>>>> per test, we also need to clean up databases at the end of a test,
> >>>> because
> >>>>>> otherwise we a) leave a lot of databases around and b) CouchDB run=
s
> >> out
> >>>> of
> >>>>>> file descriptors during the test run. The cleanup isn=E2=80=99t 10=
0% done
> yet,
> >>>> so
> >>>>>> we should check for stray databases after JS test runs until we=E2=
=80=99ve
> got
> >>>> it
> >>>>>> cleaned. One particular aspect here is when a test fails, it never
> >>>> reaches
> >>>>>> the cleanup code, so we might have to wrap everything into a
> try/catch
> >>>>>> block.
> >>>>>>
> >>>>>> - I disabled all tests that still fail and made them so they print
> the
> >>>>>> reason why they are failing, so that fixing them should be a lot
> >> easier
> >>>>>> now. Some of them print just =E2=80=9CTODO=E2=80=9D, their error c=
ause is yet to be
> >>>>>> determined.
> >>>>>>
> >>>>>> - couchdb.query() which previously executed temporary views, now
> under
> >>>> the
> >>>>>> hood creates a design doc with a random name and queries that view=
.
> >> This
> >>>>>> fixes all tests relying on temp views without having to change the=
m
> >> all
> >>>>>> individually. This is because 2.0 doesn=E2=80=99t have temp views =
anymore
> >>>> (which is
> >>>>>> a good thing).
> >>>>>>
> >>>>>> - restartServer() is no longer used (mostly).
> >>>>>>
> >>>>>> I already filed a few bugs that showed things missing in the clust=
er
> >> API
> >>>>>> that we=E2=80=99d expect to be there:
> >>>>>>
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
> >>>>>>
> >>>>>> Based on this I am ever more convinced we should get this test sui=
te
> >>>> work
> >>>>>> against the 2.0 clustered API, because we=E2=80=99ll be finding so=
me more
> >>>>>> inconsistencies that we should fix before we invite beta testers.
> >>>>>>
> >>>>>>
> >>>>>> ## Discussion:
> >>>>>>
> >>>>>> We have a whole number of tests that use the test-internal functio=
n
> >>>>>> run_on_modified_server. They use the _config API to test CouchDB i=
n
> >>>>>> different configurations from the default one. All of these tests
> now
> >>>> throw
> >>>>>> an error that _config isn=E2=80=99t available on the clustered API=
. How
> should
> >>>> we
> >>>>>> handle them?
> >>>>>>
> >>>>>> The tests are:
> >>>>>>
> >>>>>> - erlang_views.js
> >>>>>> - jsonp.js
> >>>>>> - oauth_users_db.js
> >>>>>> - oauth_users_db.js
> >>>>>> - proxyauth.js
> >>>>>> - reader_acl.js
> >>>>>> - rewrite.js
> >>>>>> - security_validation.js
> >>>>>> - show_documents.js
> >>>>>> - users_db.js
> >>>>>> - view_errors.js
> >>>>>>
> >>>>>>
> >>>>>> And a few tests aside from the proper compaction tests call _compa=
ct
> >>>> which
> >>>>>> is also not available on the cluster. What to do with them?
> >>>>>>
> >>>>>> - recreate_doc.js
> >>>>>> - and a few more that are currently disabled for other reasons,
> sorry
> >> I
> >>>>>> don=E2=80=99t have a full list at this point.
> >>>>>>
> >>>>>> * * *
> >>>>>>
> >>>>>>
> >>>>>> Bottom line: It=E2=80=99d be great if other folks here could join =
in the
> fun.
> >> No
> >>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work
> :)
> >>>>>>
> >>>>>>
> >>>>>> ## How to help:
> >>>>>>
> >>>>>> 1. Check out
> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
> >>>>>> 2. run `make clean && ./configure -c --disable-docs
> --disable-fauxton
> >> &&
> >>>>>> make`
> >>>>>> 3. `make javascript` runs the whole test suite
> >>>>>>
> >>>>>> To run the whole test suite without waiting for rebar to confirm
> that
> >> no
> >>>>>> Erlang code needs to be compiled, run `./test/javascript/run` for
> the
> >>>> whole
> >>>>>> test suite, or `./test/javascript/run all_docs` to run an individu=
al
> >>>> test
> >>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file names
> >> match
> >>>>>> accordingly).
> >>>>>>
> >>>>>> Feel free to send me PRs against the branch on
> >>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I=E2=80=
=99ll
> merge
> >>>>>> them, so the PR on apache/couchdb gets updated.
> >>>>>>
> >>>>>> I=E2=80=99m looking forward to your contributions! :)
> >>>>>>
> >>>>>> If you have any questions or have trouble getting started or just
> >> would
> >>>>>> like some hand-holding, let me know!
> >>>>>>
> >>>>>>
> >>>>>> Best
> >>>>>> Jan
> >>>>>> --
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>> --
> >>>> Professional Support for Apache CouchDB:
> >>>> http://www.neighbourhood.ie/couchdb-support/
> >>>>
> >>>>
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> http://www.neighbourhood.ie/couchdb-support/
> >>
> >>
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

--001a11428e70304a0205224df6bb--