couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garren Smith <gar...@apache.org>
Subject Re: [PROPOSAL] Gradually replace rebar with mix
Date Tue, 20 Aug 2019 12:28:12 GMT
I'm a big +1 on this. Initially, I know we would have some teething issues
getting this working. But I don't think we can stay on rebar2 forever
either. Adding in erlang dependancies is becoming increasingly more
difficult as they expect rebar3.

Cheers
Garren

On Fri, Aug 16, 2019 at 2:02 PM Ilya Khlopotov <iilyak@apache.org> wrote:

>
>
> On 2019/08/15 18:51:29, Joan Touzet <wohali@apache.org> wrote:
> > Hi Ilya,
> >
> >
> > On 2019-08-15 10:09, Ilya Khlopotov wrote:
> > > Hello,
> > >
> > > There is an https://github.com/apache/couchdb/issues/1428 issue about
> migrating to rebar3 or mix. I did an experiment to switch from rebar into
> mix and wanted to share the results. The code for experiment is here
> https://github.com/apache/couchdb/compare/master...cloudant:switch-to-mix.
> Overall I am happy with the experiment.
> > >
> > > # Requirements
> > >
> > > 1. Should be able to compile dependencies which use either rebar or
> rebar3
> > > 2. Should be able to lock dependencies to specific versions using
> references to commits
> > > 3. Should be able to call eunit test suite
> > > 4. Should be able to compile NIFs
> > > 5. Should be easy to install plugins
> > > 6. Should support dialyzer
> > > 7. Should support release managers
> > > 8. Support for raw dependencies
> > > 9. Produce source tarball suitable for offline compilation (ASF
> requirement)
> > >
> > > # How mix supports the requirements
> > >
> > > 1. Mix is able to figure out which build tool to use. There is also a
> possibility to specify build tool individually for a given dependency.
> > >   `manager: mix | rebar | rebar3 | make`. There is also support for
> calling arbitrary shell command configured via `compile: <command>`.  The
> manual step is required to install the needed tools `mix local.rebar
> --force`.
> > >    However there are means to automate that step (we can use Makefile
> or define a `bootstrap` mix alias )
> > > 2. Dependencies are locked via `mix.lock` file.
> >
> > I don't have any strong opinions one way or the other, but I did see
> > Garren asking about 'hex' instead of / in conjunction with 'mix' for
> > dependency management. Is this effort combined with his or independent?
> > I don't know the tools well enough.
>
> mix works very well with hex
>
> >
> > > 3. Eunit is supported via one of the mix plugins. Both plugins are
> very old but it should be easy to write our own (32 lines of code).
> > >  - https://github.com/dantswain/mix_eunit
> > >  - https://github.com/talentdeficit/mixunit
> > > 4. NIFs can be compiled using one of the following approaches:
> > >   - specify `rebar` as build manager for a dependency (easiest)
> >
> > Are you considering staying with rebar2 here, or moving to rebar(3)? It
> > would be nice not to have to keep supporting (and shipping!) our fork of
> > rebar2.
> I considered to stay with rebar2 for first iteration. Could you remind me
> why we are forking rebar?
> I did check that our version has no differences from upstream.
> git remote -v
> origin  https://github.com/apache/couchdb-rebar (fetch)
> origin  https://github.com/apache/couchdb-rebar (push)
> upstream        https://github.com/rebar/rebar (fetch)
> upstream        https://github.com/rebar/rebar (push)
>
> git log --oneline origin/master | head
> 0c435f3 Update master with mainline rebar (v2)
> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> 8a2aca0 travis-ci: allow Dialyzer job to fail
> 16d5dfe travis-ci: update otp versions
> 4e372a3 travis-ci: enable 20.1
> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> b05882a rebar_cover: ignore opaque type match Dialyzer warning
> 903c89d rebar_utils: fix Dialyzer warning
> b36e72b Run Dialyzer on Travis-CI
> 4bd43fe Merge pull request #645 from Juliusan/log_fix
>
> git log --oneline upstream/master | head
> b6d3094 Merge pull request #620 from tuncer/travis-dialyze
> 8a2aca0 travis-ci: allow Dialyzer job to fail
> 16d5dfe travis-ci: update otp versions
> 4e372a3 travis-ci: enable 20.1
> 73feb5f rebar_xref: ignore opaque type match Dialyzer warning
> b05882a rebar_cover: ignore opaque type match Dialyzer warning
> 903c89d rebar_utils: fix Dialyzer warning
> b36e72b Run Dialyzer on Travis-CI
> 4bd43fe Merge pull request #645 from Juliusan/log_fix
> 2276f85 Log out success message with newlines
>
> git diff origin/master upstream/master
>
> >
> > >   - call a Makefile from mix.exs. See examples:
> > >      - https://github.com/SweetIQ/expostal/blob/master/mix.exs#L9
> > >      - https://github.com/asaaki/discount.ex/blob/master/mix.exs#L6
> > >   - use https://github.com/davisp/erlang-native-compiler
> > >   - use https://github.com/blt/port_compiler
> >
> > Paul left a comment about this on the roadmap issue just today. Can you
> > respond to his comment (it was related tot dependencies-of-dependencies,
> > i.e. jiffy)
> >
> > He also raised the concern that it may force us to use only the very
> > latest Erlang versions - can you comment on this? Does the current
> > hex/mix situation require a minimum version? What version is that?
>
> I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported
> on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.
>
> > > 5. Installing plugins for mix is similar to rebar3. It is sufficient
> to just specify list of plugins in the deps in mix.exs
> > > 6. There are multiple dyalizer plugins to choose from
> > >  - https://github.com/jeremyjh/dialyxir
> > >  - https://github.com/gabrielgatu/mix-dialyzer
> >
> > I know this is a requirement for Cloudant, but today isn't for base
> > CouchDB. I would prefer not to introduce this requirement at the same
> > time as migrating build tool.
>
> The reason I mentioned it here is because we have 'make dialyze' command.
>
> > > 7. Multiple release managers are supported via mix plugins
> > >   - distillery is supported via
> https://hexdocs.pm/distillery/Mix.Releases.Plugin.html
> > >   - relx and reltools can be used via
> https://github.com/bitwalker/exrm (deprecated in favor of distillery)
> > >   - we can also roll our own mix alias using
> https://github.com/okeuday/reltool_util
> > >   - latest elixir versions have `release` command out of the box
> > > 8. Raw dependencies are supported via `compile: false` option
> > > 9. I think it is doable. In the worst case scenario we could just tar
> the content of `src` directory after calling `mix deps.get`
> > >
> > > # Warts of mix in the context of CouchDB project
> > >
> > > - CouchDB doesn't use standard elixir directories layout which caused
> include_lib directives to not work. To solve this issue absolute path to
> `<root>/src` need to be added into include path
> > >   - for dependencies managed by mix it is done via `erlc_options`
> parameter
> > >   - Unfortunately when we call out to rebar there is no way to modify
> arguments. The problem was solved via setting `ERL_COMPILER_OPTIONS`
> environment variable. Which is not very elegant solution.
> >
> > This may also cause problems on Windows, where we have very specific
> > requirements for compiler options so as to ensure the correct libraries
> > are linked for couchjs and so on.
> >
> > Have you tried compiling on that platform? ;) (Please don't make me do
> > all the work here, again...)
> >
> > > - We need to create empty `rebar.config` file for erlang applications
> which do not have `rebar.config` (all applications in couchdb repo). This
> problem can be fixed by adding mix.exs to dependencies we have control over.
> > > - The mix is slow when we have to call out to rebar. This problem can
> be fixed by adding mix.exs to dependencies we have control over.
> >
> > How much slower? Can you post a comparison of `time make couch` between
> > master and your branch?
>
> master:
>   > time bin/rebar get-deps update-deps
>   real  0m37.514s
>   user  0m1.960s
>   sys   0m3.300s
>   > time make couch
>   real  0m49.677s
>   user  0m46.910s
>   sys   0m22.210s
>
> mix:
>   > time mix do deps.get
>   real  0m24.405s
>   user  0m1.590s
>   sys   0m1.270s
>   > time mix deps.compile
>   real  3m47.968s
>   user  0m0.860s
>   sys   0m0.640s
>
> > > # Roadmap
> > >
> > > The full conversion would take some time and if we decide to go with
> mix it makes sense to do it incrementally. Possible roadmap could be:
> > > - initial PR
> > >    - offload dependency fetching to mix.exs and generate rebar.config
> so release machinery still works
> > >    - Update Makefile targets to call mix
> > >    - Choose or implement eunit plugin for mix
> > > - Support running tests individually (replacement for `make eunit
> apps= tests= suites=`)
> > > - Replace rebar plugins we use with alternative solutions
> > > - Figure out how to use dialyzer
> > > - Figure out how to use mix for preparing release
> >
> > Anything that lands on master must support release preparation. If your
> > incremental approach breaks `make dist`, you'll need to stage all of
> > your incremental changes on a feature branch and fix `make dist` before
> > merging to master. master needs to be releasable at all times.
> +1000
>
> > > Best regards,
> > > ILYA (aka iilyak)
> >
> > -Joan "release engineer hat is now off" Touzet
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message