couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peng Hui Jiang" <jian...@cn.ibm.com>
Subject Re: [RFC] On the Testing of CouchDB
Date Mon, 22 Jan 2018 01:46:38 GMT

Hi Russ,

I also see that Elixir is good way to go, compared to legacy eunit or
javascript test.

For now, I introduced one js test test case[1] for _design_docs endpoint.
Later on, I would like to convert it to elixir version.

Peng Hui (jiangphcn)

[1]:
https://github.com/apache/couchdb/pull/1110/files#diff-1e07aee7a9178c48feebcd86c5251dbc



From:	Jay Doane <jay@almery.com>
To:	dev@couchdb.apache.org
Date:	08/01/2018 02:50 PM
Subject:	Re: [RFC] On the Testing of CouchDB
Sent by:	jay.s.doane@gmail.com



Elixir's focus on positive developer experience seems like a big selling
point (among many others), so count me in.

I'd like to work on stats.js, provided nobody else has started it yet.

Jay

On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca <chewbranca@apache.org>
wrote:

> Howdy folks!
>
> The testing of CouchDB is something that has seen focus and improvements
> for the last several years, for instance migrating the etap suite to
eunit,
> and updating the JS suite to run against clusters in 2.x. There's still
> improvements to be made, and that was one of the topics of the CouchDB
dev
> summit early in the year [1].
>
> Before we go further, I want to clarify some nomenclature. I'm by no
means
> going to try and define unit testing vs integration testing vs quantum
> phase shift testing, but instead I want to focus on the distinction of
> where the testing takes place. Fundamentally, we have two places we test
> CouchDB: 1) at the Erlang VM level where we conduct assertions against
> module functions or process states; 2) at the HTTP level where we test
the
> behavior of CouchDB at the user level API. This post focuses entirely on
> the latter; that's not to say the former doesn't also merit attention,
just
> that the two are different enough that we can focus on them in isolation.
>
> So with that, let's chat about the current HTTP test suite in CouchDB.
This
> is the "JS suite" I referred to above, which is a custom built test suite
> written in Javascript and executed in the aging SpiderMonkey. The JS
suite
> has put in work for years, but it's showing it's age, and is a bit
awkward
> to work with and improve. However, I think the biggest issue with the JS
> suite is that it's utilized far less than it should be, and folks seem to
> avoid extending it or adding additional tests to it. There's been
> discussion for years about replacing said suite, but the discussions
> invariably got blocked on the bike shed of whether to rewrite the suite
in
> Javascript or Python. This thread provides a third option, with code!
>
> I started hacking on a replacement for the JS suite, this time written in
> Elixir. Overall I'm quite impressed with how it's come along, and have
some
> good examples to show. This is basically an Elixir app that has an HTTP
> client and then runs a series of tests that conduct tests against the
> CouchDB HTTP API and make assertions therein.
>
> You can find the current code in [2], and a comparison of the changes in
> [3]. The core HTTP client is only a handful of lines of codes and works
> quite well [4]. The utility functions used across all tests are located
in
> [5], and the tests themselves are in [6]. The existing test modules have
a
> 1:1 correspondence with the associated JS suite test modules, and in
> general are as direct of a port as possible.
>
> The test modules ported in their entirety or most of the way are:
>
>   * all_docs.js
>   * basics.js
>   * config.js
>   * reduce.js
>   * rewrite.js
>   * uuids.js
>   * view_collation.js
>
> Paul has dove in and is responsible for a few of those test modules and
> he's almost completed porting the replication.js suite as well. We
started
> with the hard ones first, so for the most part the rest of the ports
should
> be fairly smooth sailing.
>
> Here's an example of a very basic test:
>
> ```erlang
> defmodule WelcomeTest do
>   use CouchTestCase
>
>   test "Welcome endpoint" do
>     assert Couch.get("/").body["couchdb"] == "Welcome", "Should say
> welcome"
>   end
>
> end
>
> ```
>
>
> As you can see, the `Couch` client is very simple HTTP client with
> easy HTTP verb based methods. Let's look at a more complicated test
> for asserting we can create documents in a database:
>
>
> ```erlang
>
>   @tag :with_db
>   test "Create a document and save it to the database", context do
>     resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
> :a => 1, :b => 1}])
>     assert resp.status_code == 201, "Should be 201 created"
>     assert resp.body["id"], "Id should be present"
>     assert resp.body["rev"], "Rev should be present"
>
>     resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
>     assert resp2.body["_id"] == resp.body["id"], "Ids should match"
>     assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
>   end
>
> ```
>
>
> This is fairly straightforward code to POST a new doc, make assertions
> on the response, and then fetch the doc to make sure everything
> matches up. What I really wanted to highlight here is the `@tag
> :with_db` decorator. We can easily add custom "tags" to the tests to
> simplify setup and teardown. That `:with_db` tag does two things, it
> dynamically generates a random database name, and then takes care of
> setup/teardown for creating and deleting said database for that
> particular test. This is really useful and has been very nice to work
> with so far. We also have tag functionality in place for executing a
> test with a particular set of config options:
>
>
> ```erlang
>
>   @tag config: [
>     {"uuids", "algorithm", "utc_random"}
>   ]
>   test "utc_random uuids are roughly random" do
>     resp = Couch.get("/_uuids", query: %{:count => 1000})
>     assert resp.status_code == 200
>     uuids = resp.body["uuids"]
>
>     assert String.length(Enum.at(uuids, 1)) == 32
>
>     # Assert no collisions
>     assert length(Enum.uniq(uuids)) == length(uuids)
>
>     # Assert rough ordering of UUIDs
>     u1 = String.slice(Enum.at(uuids, 1), 0..13)
>     u2 = String.slice(Enum.at(uuids, -1), 0..13)
>     assert u1 < u2
>   end
> ```
>
>
> The tag system really simplifies a lot of the standard auxiliary
> actions needed to conduct tests.
>
>
> To test out the suite, you'll need to spin up the dev server in one
window
> with:
>
>
> ```
>
> ./dev/run --admin=adm:pass
>
> ```
>
>
> and then in another window go into the relevant CouchDB src directory and
> run:
>
>
> ```
>
> cd ~/src/couchdb/elixir_suite/
>
> mix deps.get
>
> mix test --trace
>
> ```
>
>
> The `--trace` flag makes the nice line item output per test, which I
> greatly prefer over a slew of periods. You can run an individual test
> with `mix test --trace tests/basics_test.exs`. I've pasted the output
> from running the basics suite at the bottom of this email so you can
> see what the real output looks like.
>
>
> Overall I'm quite impressed with the toolkit we've been able to put
> together in a short amount of time, and I propose we migrate fully to
> this test suite by porting all remaining JS suite tests and then
> removing the JS suite entirely. Given we've already ported most of the
> "hard suites", I think a full port is reasonable to do and just
> requires some leg work. Again, I'm impressed with how simple the
> tooling here is and how quickly we've been able to run with things,
> turns out the Elixir dev experience is actually quite nice! I hope
> others have similar opinions after diving in! Let me know what you
> think.
>
>
>
> -Russell
>
>
>
> [1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_janl_couchdb-2Dnext_issues_39&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=2X7h-oq2dK4zffTcTnVRkisZv8lzX6u4lPaNOd3xv4E&e=

> [2]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_tree_elixir-2Dsuite&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=ukdcgbNVMsAZDG4Rfkvy4Ty7tt0sVHONmxWiNNxZ4Uo&e=

> [3]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_compare_elixir-2Dsuite&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=_L7VAXIfL2cs3062JRUCa63aRd2BVhrqjuaZ8ejW_bs&e=

> [4]
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_blob_elixir-2Dsuite_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=2m-qQpRdGIB2IXwU33qA9bFE2yl-ukb4UQ2ps3tfNKE&e=

> elixir_suite/lib/couch.ex
> [5]
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_blob_elixir-2Dsuite_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=2m-qQpRdGIB2IXwU33qA9bFE2yl-ukb4UQ2ps3tfNKE&e=

> elixir_suite/test/test_helper.exs
> [6]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_couchdb_tree_elixir-2Dsuite_elixir-5Fsuite_test&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=PKZ65oA9tV05sXjYYyZUJf_d-ASaaLXiLw-gQdWPDsQ&m=KFEs1TJr_BD-8lO_QkBCmAmkSibdeftttGOftQ0IavE&s=OFNKwLRXREEHl0ba3xHseXqxp0lZRaO9aSW-bFvKL04&e=

>
>
> vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace
> test/basics_test.exs
> Excluding tags: [pending: true]
>
> BasicsTest
>   * test Session contains adm context (66.8ms)
>   * test Creating a new DB with slashes should return Location header
> (COUCHDB-411) (85.8ms)
>   * test oops, the doc id got lost in code nirwana (82.1ms)
>   * test Welcome endpoint (7.6ms)
>   * test POST doc with an _id field isn't overwritten by uuid (102.7ms)
>   * test On restart, a request for creating an already existing db can
> not override (skipped)
>   * test Creating a new DB should return location header (118.7ms)
>   * test _bulk_docs POST error when body not an object (95.0ms)
>   * test Empty database should have zero docs (161.0ms)
>   * test _all_docs POST error when multi-get is not a {'key': [...]}
> structure (104.3ms)
>   * test Regression test for COUCHDB-954 (skipped)
>   * test DELETE'ing a non-existent doc should 404 (100.0ms)
>   * test Revs info status is good (127.3ms)
>   * test PUT on existing DB should return 412 instead of 500 (97.6ms)
>   * test Database should be in _all_dbs (117.7ms)
>   * test Check for invalid document members (122.4ms)
>   * test Can create several documents (213.0ms)
>   * test Make sure you can do a seq=true option (99.1ms)
>   * test PUT doc has a Location header (skipped)
>   * test Create a document and save it to the database (116.3ms)
>   * test Created database has appropriate db info name (99.7ms)
>   * test PUT error when body not an object (89.5ms)
>   * test Simple map functions (473.0ms)
>   * test POST doc response has a Location header (117.1ms)
>
> CouchTestCase
>
>
> Finished in 3.3 seconds
> 24 tests, 0 failures, 3 skipped
>
> Randomized with seed 936284
>



Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message