Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of gs@redcometlabs.com
 designates 74.125.82.44 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\))
Subject: Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies
From: Garren Smith <garren@apache.org>
In-Reply-To: 
 <CAPaJBx60MUiMpbG_F4NTdCA_L0G8KP9d+20GRp8m5qBzG-DXUg@mail.gmail.com>
Date: Fri, 17 Jan 2014 14:00:13 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <7E9E148C-9925-4596-A232-2B775FDD0793@apache.org>
References: 
 <CAJ_m3YD=U6bUM-AAXPGxDk5eGLMy4zjH=DOB49h5x0UBXxeeDw@mail.gmail.com>
 <CAJNb-9rNP6MjvLkTtz9v7imd18YVp8RAJFOUjtCqFtBFJdVGVA@mail.gmail.com>
 <C893EA0B-0291-4835-9612-B6BD2622CF31@gmail.com>
 <CAJ_m3YC4SCsr2JNfCw_1wj0W_GBrFoyNuXXbr0E9f3f6J4SUyA@mail.gmail.com>
 <EC332A73-282A-4E00-9577-C0D57EF20A62@apache.org>
 <CAJ_m3YA-TwNqhKrt7aQGjYDzW4KbOUZ0Q8Y67e0mCz6AVuRJqQ@mail.gmail.com>
 <CAJ_m3YAaV-n2cLGWj2K+rjrEqPxknuEszEsphNPdDKuHyaesOg@mail.gmail.com>
 <CAPaJBx7hYBw33sm3inUm8r=tp_s0cUvwantsZ1-+R6gfSr9RDw@mail.gmail.com>
 <CAPaJBx60MUiMpbG_F4NTdCA_L0G8KP9d+20GRp8m5qBzG-DXUg@mail.gmail.com>
To: dev@couchdb.apache.org

I'm claiming 2nd person added!

On 17 Jan 2014, at 1:28 PM, Noah Slater <nslater@apache.org> wrote:

> Psst. A little birdy tells me that if you ask nicely, the infra folks
> will add you to the Apache GitHub org too, so you can show off your
> Apache affiliation. I was the first person added. Because I may have
> been the first to ask. ;)
>=20
> On 17 January 2014 11:56, Noah Slater <nslater@apache.org> wrote:
>> Awesome, thanks Paul.
>>=20
>> Note to all devs: if you want your contributions to CouchDB to show =
up
>> on your GitHub profile, you have to star each of the repositories.
>> (That's just how GitHub mechanics work for repo mirrors.)
>>=20
>> You can find them all here:
>>=20
>> https://github.com/apache
>>=20
>> On 17 January 2014 00:00, Paul Davis <paul.joseph.davis@gmail.com> =
wrote:
>>> New repos are up: https://git-wip-us.apache.org/repos/asf?s=3Dcouchdb
>>>=20
>>> I'm gonna go through and initialize them with history from master or
>>> one of the bigcouch and rcouch branches as appropriate.
>>>=20
>>> On Thu, Jan 16, 2014 at 2:12 PM, Paul Davis =
<paul.joseph.davis@gmail.com> wrote:
>>>> Infrastructure ticket opened: =
https://issues.apache.org/jira/browse/INFRA-7203
>>>>=20
>>>> On Thu, Jan 16, 2014 at 1:42 PM, Jan Lehnardt <jan@apache.org> =
wrote:
>>>>>=20
>>>>> On 16 Jan 2014, at 20:42 , Paul Davis =
<paul.joseph.davis@gmail.com> wrote:
>>>>>=20
>>>>>> It doesn't appear that this is objectionable to anyone. Does =
anyone
>>>>>> have an objection to us having infra/me create these repos to use =
for
>>>>>> the bigcouch/rcouch merge work? This won't affect master or =
releases
>>>>>> until those merges finish.
>>>>>=20
>>>>> no objections.
>>>>>=20
>>>>> Jan
>>>>> --
>>>>>=20
>>>>>>=20
>>>>>> On Tue, Jan 14, 2014 at 11:02 PM, Paul J Davis
>>>>>> <paul.joseph.davis@gmail.com> wrote:
>>>>>>>=20
>>>>>>>=20
>>>>>>>> On Jan 14, 2014, at 8:37 PM, Benoit Chesneau =
<bchesneau@gmail.com> wrote:
>>>>>>>>=20
>>>>>>>> On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis =
<paul.joseph.davis@gmail.com>wrote:
>>>>>>>>=20
>>>>>>>>> I've recently been having discussions about how to handle the
>>>>>>>>> repository configuration for various bits of CouchDB =
post-merge. The
>>>>>>>>> work that Benoit has been doing on the rcouch merge branch =
have also
>>>>>>>>> touched on this topic as well.
>>>>>>>>>=20
>>>>>>>>> The background for those unfamiliar is that the standard =
operating
>>>>>>>>> procedure for Erlang is to have a single Erlang application =
per
>>>>>>>>> repository and then rely on rebar to fetch each dependency.
>>>>>>>>> Traditionally in CouchDB land we've always just included the =
source to
>>>>>>>>> all applications in a single monolithic repository and =
periodically
>>>>>>>>> reimport changes from upstream dependencies.
>>>>>>>>>=20
>>>>>>>>> Recently rcouch changed from the monolithic repository to use =
external
>>>>>>>>> repositories for some dependencies. Originally the BigCouch =
used an
>>>>>>>>> even more federated scheme that had each Erlang application in =
an
>>>>>>>>> external repository (and the core couch Erlang application was =
in the
>>>>>>>>> root repository). When Bob Newson and I did the initial =
hacking on the
>>>>>>>>> BigCouch merge we pulled those external dependencies into the =
root
>>>>>>>>> repository reverting back to the large monolithic approach.
>>>>>>>>>=20
>>>>>>>>> After trying to deal with the merge and contemplating how =
various
>>>>>>>>> Erlang release things might work it's become fairly apparent =
that the
>>>>>>>>> monolithic approach is a bit constrictive. For instance, part =
of
>>>>>>>>> rebar's versioning abilities lets you tag repositories to =
generate
>>>>>>>>> versions rather than manually updating versions in source =
files.
>>>>>>>>> Another thing I've found on other projects is that having each
>>>>>>>>> application in a separate repository requires developers to =
think a
>>>>>>>>> bit more detailed about the public internal interfaces used =
through
>>>>>>>>> out the system. We've done some work to this extent already =
with
>>>>>>>>> separating source directories but forcing commits to multiple
>>>>>>>>> repositories shoots up a big red flag that maybe there's a =
high level
>>>>>>>>> of coupling between two bits of code.
>>>>>>>>>=20
>>>>>>>>> Other benefits of having the multiple repository setup is that =
its
>>>>>>>>> possible that this lends itself to being integrated with the =
proposed
>>>>>>>>> plugin system. It'd be fairly trivial to have a script that =
went and
>>>>>>>>> fetched plugins that aren't developed at Apache (as a =
./configure time
>>>>>>>>> switch type of thing). Having a system like this would also =
allow us
>>>>>>>>> to have groups focused on particular bits of development not =
have to
>>>>>>>>> concern themselves with the unrelated parts of the system.
>>>>>>>>>=20
>>>>>>>>> Given all that, I'd like to propose that we move to having a
>>>>>>>>> repository for each application/dependency that we use to =
build
>>>>>>>>> CouchDB. Each repository would be hosted on ASF infra and =
mirrored to
>>>>>>>>> GitHub as expected. This means that we could have the root =
repository
>>>>>>>>> be a simple repo that contains packaging/release/build stuff =
that
>>>>>>>>> would enable lots of the ideas offered on configurable types =
of
>>>>>>>>> release generation. I've included an initial list of =
repositories at
>>>>>>>>> the end of this email. Its basically just the apps that have =
been
>>>>>>>>> split out in either rcouch or bigcouch plus a few other bits =
from
>>>>>>>>> CouchDB master.
>>>>>>>>>=20
>>>>>>>>> I would also point out that even though our main repo would =
need to
>>>>>>>>> fetch other dependencies from the internet to build the final =
output,
>>>>>>>>> we fully intend that our release tarballs would *not* have =
this
>>>>>>>>> requirement. Ie, when we go to cut a release part of the =
process the
>>>>>>>>> RM would run would be to pull all of those dependencies before
>>>>>>>>> creating a tarball that would be wholly self contained. Given =
an
>>>>>>>>> apache-couchdb-x.y.z.tar.gz release file, there won't be a =
requirement
>>>>>>>>> to have access to the ASF git repos.
>>>>>>>>>=20
>>>>>>>>> I'm not entirely sure how controversial this is for anyone. =
For the
>>>>>>>>> most part the reactions I remember hearing were more concerned =
on
>>>>>>>>> whether the infrastructure team would allow us to use this =
sort of
>>>>>>>>> configuration. I looked yesterday and asked and apparently its
>>>>>>>>> something we can request but as always we'll want to verify =
again if
>>>>>>>>> we have consensus to move in this direction.
>>>>>>>>>=20
>>>>>>>>> Anyone have comments or flames? Right now I'm just interested =
in
>>>>>>>>> feeling out what sort of (lack of?) consensus there is on such =
a
>>>>>>>>> change. If there's general consensus I'd think we'd do a vote =
in a
>>>>>>>>> couple weeks and if that passes then start on down this road =
for the
>>>>>>>>> two merge projects and then it would become part of master =
once those
>>>>>>>>> land (as opposed to doing this to master and then attempting =
to merge
>>>>>>>>> rcouch/bigcouch onto that somehow).
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> This is a quick pass at listing what extra repositories I'd =
have
>>>>>>>>> created. Some of these applications only exist in the bigcouch =
and/or
>>>>>>>>> rcouch branches so that's where the unfamiliar application =
names are
>>>>>>>>> from. I'd also point out that the documentation and fauxton =
things are
>>>>>>>>> just on a whim in that we could decouple that development from =
the
>>>>>>>>> erlang development. I can see arguments for an against those. =
I'm much
>>>>>>>>> less concerned on that aspect than the Erlang parts that are =
directly
>>>>>>>>> affected by rebar/Erlang conventions.
>>>>>>>>>=20
>>>>>>>>>  chttpd
>>>>>>>>>  config
>>>>>>>>>  couch
>>>>>>>>>  couch_collate
>>>>>>>>>  couch_dbupdates
>>>>>>>>>  couch_httpd
>>>>>>>>>  couch_index
>>>>>>>>>  couch_mrview
>>>>>>>>>  couch_plugins
>>>>>>>>>  couch_replicator
>>>>>>>>>  documentation
>>>>>>>>>  ddoc_cache
>>>>>>>>>  ets_lru
>>>>>>>>>  fabric
>>>>>>>>>  fauxton
>>>>>>>>>  ibrowse
>>>>>>>>>  jiffy
>>>>>>>>>  mem3
>>>>>>>>>  mochiweb
>>>>>>>>>  oauth
>>>>>>>>>  rebar
>>>>>>>>>  rexi
>>>>>>>>>  snappy
>>>>>>>>>  twig
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> I also contemplated this and and I am generally +1 on this. And =
definitely
>>>>>>>> +1 to mirror them on the apache git if possible.  I have a =
couple of
>>>>>>>> comments though.
>>>>>>>>=20
>>>>>>>> Initially I also had everything separated in its own source =
repository. 1
>>>>>>>> year ago I merged back as one core repo the couchdb erlang =
applications and
>>>>>>>> put all the dependencies in the refuge repository or in the =
refuge CDN for
>>>>>>>> the spidermonkey and ICU sources.
>>>>>>>>=20
>>>>>>>> I merged back as one core repo the couchdb erlang applications =
because they
>>>>>>>> were a little too much dependant. Especially couch_httpd, =
couch_index and
>>>>>>>> couch_mrview. These applications are not yet enough by =
themselves.
>>>>>>>>=20
>>>>>>>> Imo if we split everything in  their own apps, then we should =
make sure
>>>>>>>> that couch_httpd can be used without couch_index and =
couch_mrview (which
>>>>>>>> means that "all_docs" is available in couch_httpd). Also we =
should be able
>>>>>>>> to just launch couch without any of the above. And probably =
without the
>>>>>>>> need of an ini. The couch_query_server module thing is an =
interesting case.
>>>>>>>> bigcouch is also introducing `ddoc_cache` which I am not sure =
why it is
>>>>>>>> provided as a standalone app. Does it means it can be replaced =
by another
>>>>>>>> application eventually? Why not having it simply in the  couch =
application?
>>>>>>>> Does it needs to be updated separately?
>>>>>>>>=20
>>>>>>>> Also  all our base applications should also be named spaced =
correctly so
>>>>>>>> they will be strictly identified as erlang modules:  "config" =
is too
>>>>>>>> generic, "ddoc_cache" too. Others are probably OK.
>>>>>>>>=20
>>>>>>>> There are probably other things that we could provide as apps:
>>>>>>>>=20
>>>>>>>> - couch_daemon,
>>>>>>>> - couch_js
>>>>>>>> - couch_external
>>>>>>>> - couch_stats
>>>>>>>> - couch_compaction_daemon
>>>>>>>> - couch_httpd_proxy
>>>>>>>>=20
>>>>>>>> Anyway again i'm +1 for this move, I really think it's a good =
idea.
>>>>>>>>=20
>>>>>>>> - benoit
>>>>>>>=20
>>>>>>> I agree on most of this. Roughly I see three general points.
>>>>>>>=20
>>>>>>> First, deciding on whether some things are external deps is =
definitely up for discussion. Whether couch_mrview is a different =
app/repo is not necessarily clear cut. Personally I think I over =
engineered couch_index which blurs the lines a bit. If I could wave a =
wand I'd have just couch_mrview and it'd be separate. More importantly I =
think the separate repos makes these things more apparent. The fact were =
discussing this sort of architecture thing is suggestive that it's =
forcing us to think a bit harder.
>>>>>>>=20
>>>>>>> Second is the aspect of composability. For instance the mrview =
thing to me is obviously a different repo precisely so a user could =
import couch (_core?) directly without requiring the spider monkey =
dependency. The monolithic repo doesn't allow this without some very =
non-standard tooling.
>>>>>>>=20
>>>>>>> Thirdly, app naming is always a contention. The config name was =
actually a hot code upgrade concern. We couldn't reuse couch_config =
directly at the time. And Adam was also hopeful we could the it into a =
useful non-specific config app.
>>>>>>>=20
>>>>>>> Fourthly, and related to secondly, we'll also want to look at =
splitting other apps out as necessary. The ones you listed I think =
aren't controversial it's just that no one has done it yet. My list was =
purely what existed so far without attempting to carve things up more. I =
definitely agree we should carve more in just wanted to cover consensus =
that carving is the right direction.
>>>>>>>=20
>>>>>>> Fifthly, I'm done typing on my phone. I'll fill in more thoughts =
tomorrow.
>>>>>>>=20
>>>>>=20
>>=20
>>=20
>>=20
>> --
>> Noah Slater
>> https://twitter.com/nslater
>=20
>=20
>=20
> --=20
> Noah Slater
> https://twitter.com/nslater