couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Khlopotov <iil...@apache.org>
Subject [PROPOSAL] Gradually replace rebar with mix
Date Thu, 15 Aug 2019 14:09:03 GMT
Hello,

There is an https://github.com/apache/couchdb/issues/1428 issue about migrating to rebar3
or mix. I did an experiment to switch from rebar into mix and wanted to share the results.
The code for experiment is here https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
 Overall I am happy with the experiment. 

# Requirements

1. Should be able to compile dependencies which use either rebar or rebar3
2. Should be able to lock dependencies to specific versions using references to commits
3. Should be able to call eunit test suite
4. Should be able to compile NIFs
5. Should be easy to install plugins
6. Should support dialyzer
7. Should support release managers
8. Support for raw dependencies
9. Produce source tarball suitable for offline compilation (ASF requirement)

# How mix supports the requirements

1. Mix is able to figure out which build tool to use. There is also a possibility to specify
build tool individually for a given dependency.
  `manager: mix | rebar | rebar3 | make`. There is also support for calling arbitrary shell
command configured via `compile: <command>`.  The manual step is required to install
the needed tools `mix local.rebar --force`.   
   However there are means to automate that step (we can use Makefile or define a `bootstrap`
mix alias )
2. Dependencies are locked via `mix.lock` file.
3. Eunit is supported via one of the mix plugins. Both plugins are very old but it should
be easy to write our own (32 lines of code).
 - https://github.com/dantswain/mix_eunit
 - https://github.com/talentdeficit/mixunit
4. NIFs can be compiled using one of the following approaches:
  - specify `rebar` as build manager for a dependency (easiest)
  - call a Makefile from mix.exs. See examples:
     - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
     - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
  - use https://github.com/davisp/erlang-native-compiler
  - use https://github.com/blt/port_compiler   
5. Installing plugins for mix is similar to rebar3. It is sufficient to just specify list
of plugins in the deps in mix.exs
6. There are multiple dyalizer plugins to choose from   
 - https://github.com/jeremyjh/dialyxir
 - https://github.com/gabrielgatu/mix-dialyzer
7. Multiple release managers are supported via mix plugins
  - distillery is supported via https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
  - relx and reltools can be used via https://github.com/bitwalker/exrm (deprecated in favor
of distillery)
  - we can also roll our own mix alias using https://github.com/okeuday/reltool_util
  - latest elixir versions have `release` command out of the box
8. Raw dependencies are supported via `compile: false` option
9. I think it is doable. In the worst case scenario we could just tar the content of `src`
directory after calling `mix deps.get`

# Warts of mix in the context of CouchDB project

- CouchDB doesn't use standard elixir directories layout which caused include_lib directives
to not work. To solve this issue absolute path to `<root>/src` need to be added into
include path
  - for dependencies managed by mix it is done via `erlc_options` parameter
  - Unfortunately when we call out to rebar there is no way to modify arguments. The problem
was solved via setting `ERL_COMPILER_OPTIONS` environment variable. Which is not very elegant
solution.
- We need to create empty `rebar.config` file for erlang applications which do not have `rebar.config`
(all applications in couchdb repo). This problem can be fixed by adding mix.exs to dependencies
we have control over.
- The mix is slow when we have to call out to rebar. This problem can be fixed by adding mix.exs
to dependencies we have control over.

# Roadmap 

The full conversion would take some time and if we decide to go with mix it makes sense to
do it incrementally. Possible roadmap could be:
- initial PR
   - offload dependency fetching to mix.exs and generate rebar.config so release machinery
still works
   - Update Makefile targets to call mix
   - Choose or implement eunit plugin for mix
- Support running tests individually (replacement for `make eunit apps= tests= suites=`)
- Replace rebar plugins we use with alternative solutions
- Figure out how to use dialyzer 
- Figure out how to use mix for preparing release

Best regards,
ILYA (aka iilyak)

Mime
View raw message