couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benoit Chesneau <bchesn...@gmail.com>
Subject OTP releases and autotools
Date Sat, 29 Sep 2012 08:48:15 GMT
Hi all,

This mail try to summarize the current build process and propose a
solution to integrate all requirements. Maybe it can be posted on the
wiki as a specification proposal ?

# Build summary:

## Apache CouchDB build:

- based on autotools
- install couchdb on the system
- use and require shared libraries installed on the system:
  spidermonkey, openssl, curl, icu
- require Erlang installed on the target system
- Embedding couchdb is possible but difficult.
- no hot upgrade possible
- build on linux*, unix*, bsd*, macosx, windows on i386, amd64
  platforms. Building on other platorms is possible modulo hacks on the
libraries and erlang. Windows requires a specific environment [1]


## Rcouch build

- makefile based
- use rebar [2]
- build as an OTP release [3] : the runtime is distributed as part of
  the release and custom Erlang application can be added to the build
easily, rebar is used for it.
- all libraries are fetched and built as static module that will be
  statically linked: no need to install the libraries on the target
system once the release is built. The libraries are automatically
patched and built for each supported platforms.
- build on linux*, unix*, bsd*, solaris, macosx, windows possible but need to be
  adapted
- can be embedded in other Erlang applications easily: propose a
  couch_core [4] that can be used by any other platforms
- an rcouch build can be distributed and installed on the same platform
  it has be been built for without requiring any other installation.
- can build a source release with all dependencies integrated.
- propose release templates to build custom release [5]
- hot upgrade possible


## Bigcouch build

- makefile based
- use rebar
- build an OTP release, rebar is used for it
- build couchjs using scons to find spidermonkey on the system
- libraries are not statically linked and path to find libs are set in
  rebar.config
- build on linux*, unix*, bsd*, solaris
- can be embedded in other erlang apps, but requires libs installed in
  fixed paths. Also the couch application is integrated on the release
so specific build require to fork this release.
- hot upgrade possible

# New build system proposal


This proposal is based on the following requirements:

- Possibility to embed couchdb in other erlang apps
- Create a full Erlang OTP release
- Support rebar
- Possibility to ship a static build of apache couchdb for each
  supported platforms
- Possibility to reuse installed libraries on targeted systems (good for
  distributions packagers)
- Erlang dependencies should be always available and shipped with the
  source release.

The first 2 points requires in my opinion that we can build a full couchdb
core that can be included easily in other erlangs apps. It should also
be fetched without apache-couchdb build system. The perfect way to
handle it in the Erlang world today is by using rebar and create a
`rebar.config` eventually fetching needed dependencies etc. Also some will
want to build their release statically and not care about dependencies
to install when they distribute or build their own code.

A rebar build need to know where to find shared or static libs. Some
paths can be set using env. It can be also possible to call external
scripts that will be used to build external C code and eventually set
some makefiles. rebar proposes a system of pluging to extend its
possibilities.

To use shared libraries the 2 majors and working systems today are
autotools and cmake. While cmake is interresting I think that some of us
are pefectly fluent with autotools so it may be better to use them.

I propose to mix autotools and rebar in the build process. Also I
propose to fetch erlang dependencies on build or on release instead of
having them in our sources like now. Erlang dependencies are for now:
mochiweb, oauth, ibrowse, snappy & ejson.


Imo the new build process could be the following:

- autootools used to boostrap the build: create make file and set rebar
  config files using templates.
- 2 rebar config files will be generated :
 - 1 allowing static build against libraries installed using the
  apache-couchdb-sdk, this file is generated on bootstrap. It will be
used by other erlangs code if needed.
 - The others will be created by running ./configure with needed options
- The release will be built using reltools and rebar. Using either
  dynamically generated rebar files or the static one using the
apache-couchdb-sdk
- A source release will be generated using the bootstrapped makefiled. It
  will fetch dependencies and then a signed tar.gz willl be created from
it.


The apache-couchdb-sdk is something like the android sdk or other sdk
around. It proposes a build environment for each platforms supported by
apache couchdb with all the dependencies already statically builded
(erlang, openssl, spidermonkey, icu, curl) . Something we already do for
windows and which is done somehow by rcouch too. (Note that couchbase did
the same for ios & android platforms.)

It may require some change in our current code structure. Generally an
Erlang application is organized like this:

deps/
ebin/
src/
include/
rebar.config

`deps/` is the folder where rebar put all dependencies fetches. It's
configurable now.
`ebin/` is the folder where all binaries (.beam & .app) files are put
during compilation.


When multiple apps are handled by the same project they can be
integrated eithers as dependencies or as included apps and generally the
project became:

apps/
deps/
rebar.config

See for example the couch_core repository [4] or th ebigcouch one [6].
The `apps` folder is also configurable.

For releases there are 2 ways to handled it. One is to handle it in the
same sources. So above become


apps/
deps/
rel/
rebar.config

Where the `rel` folder contains all the files needed to generate a
release. (generally a reltool.config file and a files/ folder) . This is
the way chosen by bigcouch. But generally it's better to put the release
generation in its own repository so people can integrate the core
applications they need (here couch_core) in their own releases
templates. See the `rcouch` repository for example [7]. Lot of Erlang
developers are doing this way now. My preference go for this one too but
I'm not sure how it can be done in apache. If we follow this way it will
imply to have 2 or more subprojects

1 subproject : the apache couchdb release (doc, OTP release generation)
1 subproject for the couch_core

Eventually we could split the couch_core in different apps:

- couch_core (the K/V api)
- couch_index
- couch_http
- couch_replicator

But that's can be discussed in another thread.

Voilà, hopefully this thread can be a good start for merging rcouch,
bigcouch & apache couchdb and will ease merge of the other features imo.

- benoît


[1] http://wiki.apache.org/couchdb/Installing_on_Windows
[2] https://github.com/basho/rebar/wiki
[3] http://www.erlang.org/doc/design_principles/release_structure.html#id75442
[4] https://github.com/refuge/couch_core
[5] https://github.com/refuge/rcouch_template
[6] https://github.com/cloudant/bigcouch
[7] [5] https://github.com/refuge/rcouch

Mime
View raw message