couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Samuel Newson <>
Subject Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies
Date Wed, 15 Jan 2014 00:06:34 GMT
+1 for all the reasons stated. I think we need this sooner (i.e, immediately) to proceed with
the 1843 bigcouch merge work, no?

The 1-to-1 parity will be particularly useful at helping pull in upstream changes (for mochi/ibrowse)
while a) preserving local changes (like our SOCKS5 work) and b) avoiding dependencies on third
party git hosts.


On 14 Jan 2014, at 15:22, Paul Davis <> wrote:

> I've recently been having discussions about how to handle the
> repository configuration for various bits of CouchDB post-merge. The
> work that Benoit has been doing on the rcouch merge branch have also
> touched on this topic as well.
> The background for those unfamiliar is that the standard operating
> procedure for Erlang is to have a single Erlang application per
> repository and then rely on rebar to fetch each dependency.
> Traditionally in CouchDB land we've always just included the source to
> all applications in a single monolithic repository and periodically
> reimport changes from upstream dependencies.
> Recently rcouch changed from the monolithic repository to use external
> repositories for some dependencies. Originally the BigCouch used an
> even more federated scheme that had each Erlang application in an
> external repository (and the core couch Erlang application was in the
> root repository). When Bob Newson and I did the initial hacking on the
> BigCouch merge we pulled those external dependencies into the root
> repository reverting back to the large monolithic approach.
> After trying to deal with the merge and contemplating how various
> Erlang release things might work it's become fairly apparent that the
> monolithic approach is a bit constrictive. For instance, part of
> rebar's versioning abilities lets you tag repositories to generate
> versions rather than manually updating versions in source files.
> Another thing I've found on other projects is that having each
> application in a separate repository requires developers to think a
> bit more detailed about the public internal interfaces used through
> out the system. We've done some work to this extent already with
> separating source directories but forcing commits to multiple
> repositories shoots up a big red flag that maybe there's a high level
> of coupling between two bits of code.
> Other benefits of having the multiple repository setup is that its
> possible that this lends itself to being integrated with the proposed
> plugin system. It'd be fairly trivial to have a script that went and
> fetched plugins that aren't developed at Apache (as a ./configure time
> switch type of thing). Having a system like this would also allow us
> to have groups focused on particular bits of development not have to
> concern themselves with the unrelated parts of the system.
> Given all that, I'd like to propose that we move to having a
> repository for each application/dependency that we use to build
> CouchDB. Each repository would be hosted on ASF infra and mirrored to
> GitHub as expected. This means that we could have the root repository
> be a simple repo that contains packaging/release/build stuff that
> would enable lots of the ideas offered on configurable types of
> release generation. I've included an initial list of repositories at
> the end of this email. Its basically just the apps that have been
> split out in either rcouch or bigcouch plus a few other bits from
> CouchDB master.
> I would also point out that even though our main repo would need to
> fetch other dependencies from the internet to build the final output,
> we fully intend that our release tarballs would *not* have this
> requirement. Ie, when we go to cut a release part of the process the
> RM would run would be to pull all of those dependencies before
> creating a tarball that would be wholly self contained. Given an
> apache-couchdb-x.y.z.tar.gz release file, there won't be a requirement
> to have access to the ASF git repos.
> I'm not entirely sure how controversial this is for anyone. For the
> most part the reactions I remember hearing were more concerned on
> whether the infrastructure team would allow us to use this sort of
> configuration. I looked yesterday and asked and apparently its
> something we can request but as always we'll want to verify again if
> we have consensus to move in this direction.
> Anyone have comments or flames? Right now I'm just interested in
> feeling out what sort of (lack of?) consensus there is on such a
> change. If there's general consensus I'd think we'd do a vote in a
> couple weeks and if that passes then start on down this road for the
> two merge projects and then it would become part of master once those
> land (as opposed to doing this to master and then attempting to merge
> rcouch/bigcouch onto that somehow).
> This is a quick pass at listing what extra repositories I'd have
> created. Some of these applications only exist in the bigcouch and/or
> rcouch branches so that's where the unfamiliar application names are
> from. I'd also point out that the documentation and fauxton things are
> just on a whim in that we could decouple that development from the
> erlang development. I can see arguments for an against those. I'm much
> less concerned on that aspect than the Erlang parts that are directly
> affected by rebar/Erlang conventions.
>    chttpd
>    config
>    couch
>    couch_collate
>    couch_dbupdates
>    couch_httpd
>    couch_index
>    couch_mrview
>    couch_plugins
>    couch_replicator
>    documentation
>    ddoc_cache
>    ets_lru
>    fabric
>    fauxton
>    ibrowse
>    jiffy
>    mem3
>    mochiweb
>    oauth
>    rebar
>    rexi
>    snappy
>    twig

View raw message