couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Rothbucher <sebastianrothbuc...@googlemail.com>
Subject Re: The JavaScript Test Suite
Date Tue, 17 Nov 2015 21:54:51 GMT
Hi all,

I guess the three parties last week plus 2day made a difference already -
branch 2876-js-tests starts filling and we got some bugs fixed, too. As I
see it, all replicator DB tests can be fixed more or less stereotypically.
I can pick this up Fri+ unless s/o else does it before me.

Off course, anyone should feel free to contribute to remaining tests and
any help is greatly appreciated

Anyways, there's progress - thanks again

Best
     Sebastian

On Sat, Oct 17, 2015 at 4:45 PM, Sebastian Rothbucher <
sebastianrothbucher@googlemail.com> wrote:

> Hi all,
>
> got a little further at https://github.com/janl/couchdb/pull/3/files
>
> This is it so far:
>
> # for i in test/javascript/tests/a* test/javascript/tests/b*
> test/javascript/tests/change*; dev/run -n 1 -q --with-admin-party-please
> test/javascript/run $i; done
>
> test/javascript/tests/all_docs.js   pass
>
> test/javascript/tests/attachment_names.js   pass
>
> test/javascript/tests/attachment_paths.js   pass
>
> test/javascript/tests/attachment_ranges.js   pass
>
> test/javascript/tests/attachments.js   pass
>
> test/javascript/tests/attachments_multipart.js   pass
>
> test/javascript/tests/attachment_views.js   pass
>
> test/javascript/tests/auth_cache.js   TODO  pass
>
> test/javascript/tests/basics.js   pass
>
> test/javascript/tests/batch_save.js   pass
>
> test/javascript/tests/bulk_docs.js   pass
>
> test/javascript/tests/changes.js   pass
>
>
> It uncovers some problems with latest=... in the URL and Last event ID a
> header param that might be worth a ticket
>
>
> We're rolling... and still some work 2 do ;-)
>
>
>     Sebastian
>
>
> On Wed, Oct 14, 2015 at 12:23 PM, Jan Lehnardt <jan@apache.org> wrote:
>
>> I’ve run the 1.x test suite against an -n1 cluster and it doesn’t improve
>> things
>> significantly, although I haven’t looked at database deletes, as I’ve got
>> these
>> covered for now.
>>
>> So there is still some work that needs doing :)
>>
>> Best
>> Jan
>> --
>>
>> > On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
>> >
>> > Started with
>> >
>> > python dev/run -n 1 --with-admin-party-please &
>> >
>> > and export COUCH_HOST='http://127.0.0.1:15984'
>> >
>> >
>> > On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
>> >
>> >>
>> >>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
>> >>>
>> >>>> - instead of fixed test db names like `test_suite_db` that are used
>> in
>> >>> almost every test,
>> >>>> we now generate a random database name for each test. The JS tests
>> >>> expected db
>> >>>> deletion to be synchronous, and that’s no longer the case in 2.0.
>> >> Instead
>> >>> of
>> >>>> error-prone polling to wait for db deletion, we just use new db
for
>> >> every
>> >>> test.
>> >>>> An added benefit now is that (some) tests can run in parallel.
>> >>>
>> >>> Little confused by this, PouchDB uses consistent names in its test
>> suite
>> >>> (we had random
>> >>> ones for a while and moved away), we very much depend on the database
>> >> being
>> >>> deleted
>> >>> on the request returning and dont seem to have any problems with it
>> >> against
>> >>> master.
>> >>
>> >> Does the PouchDB test suite runs against the 5986 port, or with an n=1
>> >> cluster?
>> >> This is with the default n=3 cluster against 5986.
>> >>
>> >> Best
>> >> Jan
>> >> --
>> >>
>> >>>
>> >>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
>> >>>
>> >>>>
>> >>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
>> >>>> sebastianrothbucher@googlemail.com> wrote:
>> >>>>>
>> >>>>> Hi Jan,
>> >>>>>
>> >>>>> I transferred some of my findings to your branch - pls. find
a PR
>> for
>> >> the
>> >>>>> first few tests submitted 2 your branch. I put in TODOs where
either
>> >>>> there
>> >>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853)
or we
>> have
>> >>>> to
>> >>>>> do some more work (like making config available).
>> >>>>
>> >>>> Excellent, thanks! :)
>> >>>>
>> >>>>
>> >>>>> Will from the Pouch team had the great idea of using -n 1 to
>> mitigate
>> >>>>> Heisenbugs due to distribution. I adopted it and it did work
quite
>> >> well.
>> >>>> It
>> >>>>> might make sense to put this into the dev/run call as a general
>> rule,
>> >>>> esp.
>> >>>>> when checking for 201 (VS 202) status codes. Probably you know
more
>> on
>> >>>>> that.
>> >>>>
>> >>>> We should carefully evaluate this :)
>> >>>>
>> >>>>
>> >>>>> Anyways: I think we have to keep using _config on the local
port as
>> >> many
>> >>>>> tests (like auth tests) depend on changed configurations. BTW:
w/
>> -n 1
>> >>>> it's
>> >>>>> only one call, but calling 3 _config(s) migt also work.
>> >>>>> Hopefully (I just arrived at some test w/ that recently) we
can
>> spare
>> >>>>> ourselves the _compact thing. I think we'd have to compact every
>> single
>> >>>>> shard via local port, right? for the first tests, we can go
w/out
>> as it
>> >>>>> seems. Maybe we should stick to that for a while.
>> >>>>
>> >>>> My concern with this is that as far as I care, the local port is
>> >> irrelevant
>> >>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but
the
>> >> cluster
>> >>>> port is our publicised API for 2.0 and it’s the one that replaces
the
>> >> 1.x
>> >>>> API for people upgrading. So having any tests that run on the local
>> port
>> >>>> of 2.0 don’t help us in any way (that’s not to say it still
useful to
>> >> have
>> >>>> tests against them for the moment).
>> >>>>
>> >>>> Many of the _config-dependent tests substitute out the _users or
>> >>>> _replicator
>> >>>> databases, maybe we can find a workaround for those by just using
the
>> >> stock
>> >>>> databases.
>> >>>>
>> >>>> For the other tests, maybe we split them out into their own “test
>> >> suites”
>> >>>> that start their own test cluster in the configuration they need
and
>> >> shut
>> >>>> it down afterwards.
>> >>>>
>> >>>> Best
>> >>>> Jan
>> >>>> --
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> Looking fw 2 your thoughts
>> >>>>>
>> >>>>> Best
>> >>>>>  Sebastian
>> >>>>>
>> >>>>> P.S::  Maybe we can adopt some stuff from
>> >>>>>
>> >>>>
>> >>
>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
>> >>>>>
>> >>>>> P.P.S.: Will keep working on it, just not in the next few days,
>> >>>>> unfortunately ;-(
>> >>>>>
>> >>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
>> wrote:
>> >>>>>
>> >>>>>> (I need your help deciding what to do with a bunch of tests,
jump
>> to
>> >>>>>> “Discussion” below if you don’t care about the details
before.)
>> >>>>>>
>> >>>>>> * * *
>> >>>>>>
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> I spent some more time getting the 1.x JavaScript tests
running on
>> the
>> >>>>>> cluster port of 2.0. WIP here:
>> >>>> https://github.com/apache/couchdb/pull/354
>> >>>>>> — I’ve gotten a lot further, but there is still a lot
to do and
>> >> everyone
>> >>>>>> here can help, I’ve posted instructions on how to do so
further
>> down.
>> >> No
>> >>>>>> Erlang knowledge needed, this is ideal for folks who want
to start
>> >> with
>> >>>>>> contributing to CouchDB.
>> >>>>>>
>> >>>>>>
>> >>>>>> Notable changes:
>> >>>>>>
>> >>>>>> - instead of fixed test db names like `test_suite_db` that
are
>> used in
>> >>>>>> almost every test, we now generate a random database name
for each
>> >> test.
>> >>>>>> The JS tests expected db deletion to be synchronous, and
that’s no
>> >>>> longer
>> >>>>>> the case in 2.0. Instead of error-prone polling to wait
for db
>> >>>> deletion, we
>> >>>>>> just use new db for every test. An added benefit now is
that (some)
>> >>>> tests
>> >>>>>> can run in parallel.
>> >>>>>>
>> >>>>>> - tests now clean up after themselves (incomplete): tests
>> previously
>> >>>> only
>> >>>>>> collectively used three or four databases, and deleted and
>> re-created
>> >>>> them
>> >>>>>> before each test was run. Now with the change of creating
a new
>> >> database
>> >>>>>> per test, we also need to clean up databases at the end
of a test,
>> >>>> because
>> >>>>>> otherwise we a) leave a lot of databases around and b) CouchDB
runs
>> >> out
>> >>>> of
>> >>>>>> file descriptors during the test run. The cleanup isn’t
100% done
>> yet,
>> >>>> so
>> >>>>>> we should check for stray databases after JS test runs until
we’ve
>> got
>> >>>> it
>> >>>>>> cleaned. One particular aspect here is when a test fails,
it never
>> >>>> reaches
>> >>>>>> the cleanup code, so we might have to wrap everything into
a
>> try/catch
>> >>>>>> block.
>> >>>>>>
>> >>>>>> - I disabled all tests that still fail and made them so
they print
>> the
>> >>>>>> reason why they are failing, so that fixing them should
be a lot
>> >> easier
>> >>>>>> now. Some of them print just “TODO”, their error cause
is yet to be
>> >>>>>> determined.
>> >>>>>>
>> >>>>>> - couchdb.query() which previously executed temporary views,
now
>> under
>> >>>> the
>> >>>>>> hood creates a design doc with a random name and queries
that view.
>> >> This
>> >>>>>> fixes all tests relying on temp views without having to
change them
>> >> all
>> >>>>>> individually. This is because 2.0 doesn’t have temp views
anymore
>> >>>> (which is
>> >>>>>> a good thing).
>> >>>>>>
>> >>>>>> - restartServer() is no longer used (mostly).
>> >>>>>>
>> >>>>>> I already filed a few bugs that showed things missing in
the
>> cluster
>> >> API
>> >>>>>> that we’d expect to be there:
>> >>>>>>
>> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>> >>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>> >>>>>>
>> >>>>>> Based on this I am ever more convinced we should get this
test
>> suite
>> >>>> work
>> >>>>>> against the 2.0 clustered API, because we’ll be finding
some more
>> >>>>>> inconsistencies that we should fix before we invite beta
testers.
>> >>>>>>
>> >>>>>>
>> >>>>>> ## Discussion:
>> >>>>>>
>> >>>>>> We have a whole number of tests that use the test-internal
function
>> >>>>>> run_on_modified_server. They use the _config API to test
CouchDB in
>> >>>>>> different configurations from the default one. All of these
tests
>> now
>> >>>> throw
>> >>>>>> an error that _config isn’t available on the clustered
API. How
>> should
>> >>>> we
>> >>>>>> handle them?
>> >>>>>>
>> >>>>>> The tests are:
>> >>>>>>
>> >>>>>> - erlang_views.js
>> >>>>>> - jsonp.js
>> >>>>>> - oauth_users_db.js
>> >>>>>> - oauth_users_db.js
>> >>>>>> - proxyauth.js
>> >>>>>> - reader_acl.js
>> >>>>>> - rewrite.js
>> >>>>>> - security_validation.js
>> >>>>>> - show_documents.js
>> >>>>>> - users_db.js
>> >>>>>> - view_errors.js
>> >>>>>>
>> >>>>>>
>> >>>>>> And a few tests aside from the proper compaction tests call
>> _compact
>> >>>> which
>> >>>>>> is also not available on the cluster. What to do with them?
>> >>>>>>
>> >>>>>> - recreate_doc.js
>> >>>>>> - and a few more that are currently disabled for other reasons,
>> sorry
>> >> I
>> >>>>>> don’t have a full list at this point.
>> >>>>>>
>> >>>>>> * * *
>> >>>>>>
>> >>>>>>
>> >>>>>> Bottom line: It’d be great if other folks here could join
in the
>> fun.
>> >> No
>> >>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP
API-work
>> :)
>> >>>>>>
>> >>>>>>
>> >>>>>> ## How to help:
>> >>>>>>
>> >>>>>> 1. Check out
>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>> >>>>>> 2. run `make clean && ./configure -c --disable-docs
>> --disable-fauxton
>> >> &&
>> >>>>>> make`
>> >>>>>> 3. `make javascript` runs the whole test suite
>> >>>>>>
>> >>>>>> To run the whole test suite without waiting for rebar to
confirm
>> that
>> >> no
>> >>>>>> Erlang code needs to be compiled, run `./test/javascript/run`
for
>> the
>> >>>> whole
>> >>>>>> test suite, or `./test/javascript/run all_docs` to run an
>> individual
>> >>>> test
>> >>>>>> (all_docs runs test/javascript/tests/all_docs.js, other
file names
>> >> match
>> >>>>>> accordingly).
>> >>>>>>
>> >>>>>> Feel free to send me PRs against the branch on
>> >>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip,
I’ll
>> merge
>> >>>>>> them, so the PR on apache/couchdb gets updated.
>> >>>>>>
>> >>>>>> I’m looking forward to your contributions! :)
>> >>>>>>
>> >>>>>> If you have any questions or have trouble getting started
or just
>> >> would
>> >>>>>> like some hand-holding, let me know!
>> >>>>>>
>> >>>>>>
>> >>>>>> Best
>> >>>>>> Jan
>> >>>>>> --
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>> --
>> >>>> Professional Support for Apache CouchDB:
>> >>>> http://www.neighbourhood.ie/couchdb-support/
>> >>>>
>> >>>>
>> >>
>> >> --
>> >> Professional Support for Apache CouchDB:
>> >> http://www.neighbourhood.ie/couchdb-support/
>> >>
>> >>
>>
>> --
>> Professional Support for Apache CouchDB:
>> http://www.neighbourhood.ie/couchdb-support/
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message