couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Khlopotov" <iil...@ca.ibm.com>
Subject Re: [PROPOSAL] Improve testing infrastructure of couchdb
Date Thu, 01 Oct 2015 22:24:14 GMT

With elixir things way easier for sure. Most of the problems wouldn't exist
with elixir (for example scope annotations or chaining setups).

https://github.com/batate/shouldi
https://github.com/josephwilk/amrita
https://github.com/pinfieldharm/test_times - track test times when running
mix test
https://github.com/antp/kovacs - monitors changes in directories and run
tests
https://github.com/meadsteve/white-bread - Story BDD tool for elixir using
gherkin
https://github.com/victorolinasc/junit-formatter - Reporter for jenkins
https://github.com/sproutapp/pavlov - BDD elixir based DSL
https://github.com/antonmi/espec - BDD which it is not a wrapper on top of
exunit

None of them perfect though. Even though going with elixir would provide
technological advantage. It would impose a steep learning curve on
developers.

BR,
ILYA



From:	Alexander Shorin <kxepal@gmail.com>
To:	"dev@couchdb.apache.org" <dev@couchdb.apache.org>
Date:	2015/10/01 02:29 PM
Subject:	Re: [PROPOSAL] Improve testing infrastructure of couchdb



Hi Iliya,

I'll add my 5 cents.

Current test suite isn't perfect. It's actually the same what we have
with etap + few cosmetic improvements and isolated on resource usage.
It must be improved, no doubt and any move on this way is all welcome!

Most of our tests issues are actually not about fight with bugs in
code but about fight with bugs in tests. This takes a lot of resources
for almost no profit. Sometimes it's much more easy to test feature
manually with raw curl rather than write some erlang/js test.

I think we should think at first place about how to stop continuously
fix our tests and focus more on our code. If there would be special
app for testing, I would like to see how it provides high level
interface to hide all the boring routines that we have now.

I also think that we should use diverse tools for testing, each for
their scope. Eunit is very simple and good for unit testing, but for
integration tests it completely horrible and allows to fall into
unknown state when all following tests will crash with some weird
reason. We could use commontests for functional and integration
testing. We could try triq/excheck/eqc[1] for property testing things,
even for developers-only this still would be helpful.

We can even try to look on elixir with espec and other tools[2] it
provided. Why not, if it will simplify testing process?

Developers should write more test, but they wouldn't white this
process will take more time, resources, lines of code. Testing should
be fun. Now it's a big pain and constant repeat of our selfs.

If couch_tests will provide all the set of tools to make testing
simpler, focus on matter and don't waste a time on unimportant
details, I'm +1 for this.

[1]: quite good example here:
http://www.quviq.com/adding-a-github-repository-to-quickcheck-ci/
[2]: https://github.com/h4cc/awesome-elixir#testing



--
,,,^..^,,,


On Thu, Oct 1, 2015 at 11:32 PM, Ilya Khlopotov <iilyak@ca.ibm.com> wrote:
>
>
> Hi,
>
> I would like to submit a proposal to improve the way we test couchdb.
Here
> is the link to rendered version:
>
https://rawgit.com/iilyak/e65b4ddf8a46416f84be/raw/4c8dff91d51fcfd6e5a7f65b7e9b0f8c96ee3a61/proposal.html

>
> Bellow is the source of the proposal
> ------------------------------
>
> # Testing couchdb
>
> It is quite hard to test couchdb currently. The main difficulties are
> related
> to complex setup functions and weak isolation of test cases. There are
some
> functions in `test_util` module which simplify setup a little bit but not
> by much.
> The purpose of this proposal is to define some requirements for testing
> infrastructure and proposing a solution which satisfy most of these
> requirements.
>
> ## Why is it so hard to write certain kinds of tests?
>
> - Setup / teardown overhead.
> - Difficult-to-reproduce failure modes.
> - Tests which require clusters rather than single nodes.
> - Tests for issues that only manifest at scale (e.g. >100 nodes).
> - Verifying/manipulating aspects of internal state during / after a test.
> - PRs for functional changes are not isolated to single repositories
which
> makes integration with CI (required to enforce any
must-have-passing-tests
> rules we might want) or manual testing of multi-repo changes difficult.
> - Code which is difficult to test due to coupling of IO with pure logic.
>
> ## What might make things easier?
>
> - A high level API for creating test fixtures (e.g. nodes/clusters in
> specific failure states but also other things like configuration files,
> logs, etc).
> - A high level API for manipulating node/cluster state (e.g. stop/start
> node, simulate packet loss, partition one or more nodes).
> - Lower-level constructs for directly modifying code under-test e.g.:
>   - forcing a particular error to be thrown at a particular time
>   - tapping into the output of a logger
>   - tapping into metrics collector
>   - being able to tell that specified function has been called and
possibly
> get arguments passed to that function
>   - being able to store terms into temporary storage during the execution
> of a test case
>   - facilities to group tests
>   - ability to suppress logging or individual log messages
>   - be able to run same behaviour tests against different implementations
>   - be able to run same test suites against different configurations
> - Tooling for testing specific branches of the sub-repositories (e.g.
> https://cloudup.com/cOgxRPbt9aP).
> - A manifest repo which links to all proposed changes that span multiple
> repos (e.g. https://github.com/iilyak/couchdb-manifest).
> - Refactoring to separate IO from pure logic.
> - Adding function specifications and increased use of dialyzer as part of
> the CI chain.
> - Track test suite execution time to detect performance degradation.
> - Refactoring to be able to specify names for named processes
>
> # Proposal
>
> - Create `couch_test` app which would contain helper functions to make
> eunit
> easier to use for couchdb testing. It would include solutions for some of
> the identified earlier problems.
> - Write new tests using `cdt:setup` `cdt:make_casses`
> - Use [riak_test's intercepts](
> https://github.com/basho/riak_test/tree/master/intercepts)
> - Update Makefile to use `src/couch_test/bin/cdt ci`
> - Exact implementation might be different from the design bellow
> - Improve as we go
>
>
> # couch_test (cdt) design
>
>     couch_test/
>       +-- include/
>         +-- intercept.hrl
>         +-- cdt.hrl
>       +-- intercepts/
>       +-- setups/ - Would contain setup/teardown code for reuse
>       +-- src
>         +-- intercept.erl
>         +-- cdt.erl
>         +-- combinatorics.erl
>       +-- bin/
>         +-- cdt
>       +-- rebar.config
>       +-- etc/
>         +-- cdt.conf.example
>         +-- local.conf.example
>         +-- test_cluster.conf.example
>
> Just to illustrate the idea here is the list of commands for `cdt`
> (eventually).
>
> `cdt --help`
>
>   - cdt ci
>   - cdt all
>   - cdt unit
>   - cdt integration
>   - cdt system
>   - cdt perf
>   - cdt props
>   - cdt scale -nodes 200
>   - cdt vmargs -period 3600000 -nodes 3
>     - run VM with different vmargs (generated using powerset)
>       to find out best combination of options and flags.
>       We restart VM with next set of options after 1 hour
>   - cdt -file file.erl
>   - cdt -module module
>   - cdt -test module:fun
>   - cdt mixed -nodes 10 -old versionA -new versionB
>
>
> ## setup modules
>
>     -module(cdt_chttpd_setup).
>
>     -export([chttpd/4]).
>
>     chttpd(A, B, C, D) ->
>         {setup(A, B, C, D), teardown()}.
>
>     setup(A, B, C, D) ->
>         fun(Ctx) ->
>             %% use A and B to setup chttpd
>             %% store C and D in context for latter use in tests
>             NewCtx = update_ctx(Ctx, C, D),
>             {?MODULE, NewCtx}
>         end.
>
>     teardown() ->
>         fun(Ctx) ->
>            %% Clean up the things using Ctx to find out what to do
>            {?MODULE, Ctx}
>         end.
>
> Setups could be composed into a chain
>
>     -module(my_unit_test).
>
>     -import_lib("cdt.hrl").
>
>     %% Imports are optional
>     -import([cdt_chttpd_setup, [chttpd/4]]).
>     -import([cdt_couch_setup, [couch/3]]).
>     -import([cdt_cluster_setup, [cluster/1]]).
>     -import([cdt_fault_setup, [disconnect/2, drop_packets/3]]).
>
>     setup(Type) ->
>         Chain = [
>             couch(Type, 1, 2),
>             chttpd(backdoor, foo, bar, baz),
>             cluster(3),
>             disconnect(1, 2),
>             drop_packets(2, 3, 30) %% 30% of packets to drop between db2
> and db3 nodes
>         ],
>         Args = [],
>         Opts = [],
>         cdt:setup(Chain, Args, Opts).
>
>     teardown(_Type, Ctx) ->
>         cdt:teardown(Ctx).
>
> ## Injecting networking problems
>
> Since a crossplatform solution is required it is better to use something
> erlang
> based for fault injection. We could extend [epmdpxy](
> https://github.com/dergraf/epmdpxy)
> to simulate latency or connectivity problems. It should be extended in
such
> a
> way to be able to selectively induce problems between specified nodes
> (without affecting the communication between test master and test
slaves).
> In this case nodes should be started using:
>
>     ERL_EPMD_PORT=43690 erl
>
> ## Tapping to logger
>
> Don't log or produce noisy output by default.
> However should be able to control verbosity. It is also possible to split
> the output.
>
>   - stdout: Errors and output from eunit
>   - file per test suite run: For detailed logging
>
> In some cases should be able to:
>
>  - suppress log message
>  - check error message produced by logger
>
> The easiest way to achieve both goals is using permanent intercept for
> `couch_log.erl`.
> Another approach could be a special `couch_log` backend.
>
> ## Fixtures
>
> Store fixtures in `tests/fixtures` of the applications we are testing.
> We also might have some common fixtures in `couch_test/fixtures`.
> All fixtures should be templates.
> `couch_test` app would have some helpers to find and include the
fixtures.
>
> It would be helpful to support following types of fixtures:
>
>  - module
>  - file
>  - data structure
>     - `<name>.script` - similar to `file:script` but with template
> rendering
>     - `<name>` - similar to `file:consult` but with template rendering
>
>
> ## Grouping test cases
>
> Use list of lists as a test name to determine if grouping is needed.
> For example:
>
>     apply_options_test_() ->
>         Funs = [fun ensure_apply_is_called/2],
>         Cases = combinatorics:powerset([pipe, concurrent]),
>         cdt:make_cases(
>             ["apply options tests", "Apply with options: ~p"],
>             fun setup/1, fun teardown/2,
>             Cases, Funs).
>
> This would generate following test cases
>
>     {
>         "apply options tests",
>         [
>             {
>                 "Apply with options: []",
>                 [
>                     {
>                         foreachx, fun setup/1, fun teardown/2,
>                         [
>                             {[], fun ensure_apply_is_called/2}
>                         ]
>                     }
>                 ]
>             },
>             {
>                 "Apply with options: [pipe]",
>                 [
>                     {
>                         foreachx, fun setup/1, fun teardown/2,
>                         [
>                             {[pipe], fun ensure_apply_is_called/2}
>                         ]
>                     }
>                 ]
>             },
>             {
>                 "Apply with options: [concurent]",
>                 [
>                     {
>                         foreachx, fun setup/1, fun teardown/2,
>                         [
>                             {[concurent], fun ensure_apply_is_called/2}
>                         ]
>                     }
>                 ]
>             },
>             {
>                 "Apply with options: [pipe, concurrent]",
>                 [
>                     {
>                         foreachx, fun setup/1, fun teardown/2,
>                         [
>                             {[pipe, concurrent], fun
> ensure_apply_is_called/2}
>                         ]
>                     }
>                 ]
>             },
>         ]
>
>
> ## Tests annotations
>
> In order to distinguish kinds of tests we would need to annotate test
> cases.
> We could use one of the following in order of my personal preference
> (any other ideas?):
>
> 1. Implement parse transform using `merl` to support annotations:
>
>         -scope([integration, unit, cluster]).
>         my_tests_() - >
>             ok.
>
> 2. Split different kinds of tests into different modules and maybe
>   keep them in different directories
> 3. Pass scope to `cdt:make_cases`.
> 4. Introduce naming convention for a test name
>
>    - `i_my_integration_tests_() -> ok`.
>    - `u_my_unit_tests_() -> ok`.
>
> 5. Have a module where we add every test case into approporiate scope.
>
> --------------------------------------------
>
> Best regards,
> ILYA




Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message