couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@apache.org>
Subject Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies
Date Fri, 17 Jan 2014 19:44:30 GMT
Done. Edit away! (Thanks!)

On 17 January 2014 20:28, Mutton, James <jmutton@akamai.com> wrote:
> JamesMutton
>
> </JamesM>
>
>
>
>
> On 1/17/14 11:09 AM, "Noah Slater" <nslater@apache.org> wrote:
>
>>Good plan. What's your wiki username? I'll add you.
>>
>>On 17 January 2014 20:01, Mutton, James <jmutton@akamai.com> wrote:
>>> Might I suggest that this sounds like good info to document on the wiki
>>> for committers getting started.  I'd add it but I'm not in the
>>>allow-list.
>>> :)
>>>
>>> </JamesM>
>>>
>>>
>>>
>>>
>>> On 1/17/14 4:00 AM, "Garren Smith" <garren@apache.org> wrote:
>>>
>>>>I'm claiming 2nd person added!
>>>>
>>>>On 17 Jan 2014, at 1:28 PM, Noah Slater <nslater@apache.org> wrote:
>>>>
>>>>> Psst. A little birdy tells me that if you ask nicely, the infra folks
>>>>> will add you to the Apache GitHub org too, so you can show off your
>>>>> Apache affiliation. I was the first person added. Because I may have
>>>>> been the first to ask. ;)
>>>>>
>>>>> On 17 January 2014 11:56, Noah Slater <nslater@apache.org> wrote:
>>>>>> Awesome, thanks Paul.
>>>>>>
>>>>>> Note to all devs: if you want your contributions to CouchDB to show
>>>>>>up
>>>>>> on your GitHub profile, you have to star each of the repositories.
>>>>>> (That's just how GitHub mechanics work for repo mirrors.)
>>>>>>
>>>>>> You can find them all here:
>>>>>>
>>>>>> https://github.com/apache
>>>>>>
>>>>>> On 17 January 2014 00:00, Paul Davis <paul.joseph.davis@gmail.com>
>>>>>>wrote:
>>>>>>> New repos are up: https://git-wip-us.apache.org/repos/asf?s=couchdb
>>>>>>>
>>>>>>> I'm gonna go through and initialize them with history from master
or
>>>>>>> one of the bigcouch and rcouch branches as appropriate.
>>>>>>>
>>>>>>> On Thu, Jan 16, 2014 at 2:12 PM, Paul Davis
>>>>>>><paul.joseph.davis@gmail.com> wrote:
>>>>>>>> Infrastructure ticket opened:
>>>>>>>>https://issues.apache.org/jira/browse/INFRA-7203
>>>>>>>>
>>>>>>>> On Thu, Jan 16, 2014 at 1:42 PM, Jan Lehnardt <jan@apache.org>
>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>> On 16 Jan 2014, at 20:42 , Paul Davis
>>>>>>>>><paul.joseph.davis@gmail.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>> It doesn't appear that this is objectionable to anyone.
Does
>>>>>>>>>>anyone
>>>>>>>>>> have an objection to us having infra/me create these
repos to use
>>>>>>>>>>for
>>>>>>>>>> the bigcouch/rcouch merge work? This won't affect
master or
>>>>>>>>>>releases
>>>>>>>>>> until those merges finish.
>>>>>>>>>
>>>>>>>>> no objections.
>>>>>>>>>
>>>>>>>>> Jan
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 14, 2014 at 11:02 PM, Paul J Davis
>>>>>>>>>> <paul.joseph.davis@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Jan 14, 2014, at 8:37 PM, Benoit Chesneau
>>>>>>>>>>>><bchesneau@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis
>>>>>>>>>>>><paul.joseph.davis@gmail.com>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I've recently been having discussions
about how to handle the
>>>>>>>>>>>>> repository configuration for various
bits of CouchDB
>>>>>>>>>>>>>post-merge. The
>>>>>>>>>>>>> work that Benoit has been doing on the
rcouch merge branch
>>>>>>>>>>>>>have
>>>>>>>>>>>>>also
>>>>>>>>>>>>> touched on this topic as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The background for those unfamiliar is
that the standard
>>>>>>>>>>>>>operating
>>>>>>>>>>>>> procedure for Erlang is to have a single
Erlang application
>>>>>>>>>>>>>per
>>>>>>>>>>>>> repository and then rely on rebar to
fetch each dependency.
>>>>>>>>>>>>> Traditionally in CouchDB land we've always
just included the
>>>>>>>>>>>>>source to
>>>>>>>>>>>>> all applications in a single monolithic
repository and
>>>>>>>>>>>>>periodically
>>>>>>>>>>>>> reimport changes from upstream dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Recently rcouch changed from the monolithic
repository to use
>>>>>>>>>>>>>external
>>>>>>>>>>>>> repositories for some dependencies. Originally
the BigCouch
>>>>>>>>>>>>>used an
>>>>>>>>>>>>> even more federated scheme that had each
Erlang application in
>>>>>>>>>>>>>an
>>>>>>>>>>>>> external repository (and the core couch
Erlang application was
>>>>>>>>>>>>>in the
>>>>>>>>>>>>> root repository). When Bob Newson and
I did the initial
>>>>>>>>>>>>>hacking
>>>>>>>>>>>>>on the
>>>>>>>>>>>>> BigCouch merge we pulled those external
dependencies into the
>>>>>>>>>>>>>root
>>>>>>>>>>>>> repository reverting back to the large
monolithic approach.
>>>>>>>>>>>>>
>>>>>>>>>>>>> After trying to deal with the merge and
contemplating how
>>>>>>>>>>>>>various
>>>>>>>>>>>>> Erlang release things might work it's
become fairly apparent
>>>>>>>>>>>>>that the
>>>>>>>>>>>>> monolithic approach is a bit constrictive.
For instance, part
>>>>>>>>>>>>>of
>>>>>>>>>>>>> rebar's versioning abilities lets you
tag repositories to
>>>>>>>>>>>>>generate
>>>>>>>>>>>>> versions rather than manually updating
versions in source
>>>>>>>>>>>>>files.
>>>>>>>>>>>>> Another thing I've found on other projects
is that having each
>>>>>>>>>>>>> application in a separate repository
requires developers to
>>>>>>>>>>>>>think a
>>>>>>>>>>>>> bit more detailed about the public internal
interfaces used
>>>>>>>>>>>>>through
>>>>>>>>>>>>> out the system. We've done some work
to this extent already
>>>>>>>>>>>>>with
>>>>>>>>>>>>> separating source directories but forcing
commits to multiple
>>>>>>>>>>>>> repositories shoots up a big red flag
that maybe there's a
>>>>>>>>>>>>>high
>>>>>>>>>>>>>level
>>>>>>>>>>>>> of coupling between two bits of code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Other benefits of having the multiple
repository setup is that
>>>>>>>>>>>>>its
>>>>>>>>>>>>> possible that this lends itself to being
integrated with the
>>>>>>>>>>>>>proposed
>>>>>>>>>>>>> plugin system. It'd be fairly trivial
to have a script that
>>>>>>>>>>>>>went and
>>>>>>>>>>>>> fetched plugins that aren't developed
at Apache (as a
>>>>>>>>>>>>>./configure time
>>>>>>>>>>>>> switch type of thing). Having a system
like this would also
>>>>>>>>>>>>>allow us
>>>>>>>>>>>>> to have groups focused on particular
bits of development not
>>>>>>>>>>>>>have to
>>>>>>>>>>>>> concern themselves with the unrelated
parts of the system.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Given all that, I'd like to propose that
we move to having a
>>>>>>>>>>>>> repository for each application/dependency
that we use to
>>>>>>>>>>>>>build
>>>>>>>>>>>>> CouchDB. Each repository would be hosted
on ASF infra and
>>>>>>>>>>>>>mirrored to
>>>>>>>>>>>>> GitHub as expected. This means that we
could have the root
>>>>>>>>>>>>>repository
>>>>>>>>>>>>> be a simple repo that contains packaging/release/build
stuff
>>>>>>>>>>>>>that
>>>>>>>>>>>>> would enable lots of the ideas offered
on configurable types
>>>>>>>>>>>>>of
>>>>>>>>>>>>> release generation. I've included an
initial list of
>>>>>>>>>>>>>repositories at
>>>>>>>>>>>>> the end of this email. Its basically
just the apps that have
>>>>>>>>>>>>>been
>>>>>>>>>>>>> split out in either rcouch or bigcouch
plus a few other bits
>>>>>>>>>>>>>from
>>>>>>>>>>>>> CouchDB master.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would also point out that even though
our main repo would
>>>>>>>>>>>>>need to
>>>>>>>>>>>>> fetch other dependencies from the internet
to build the final
>>>>>>>>>>>>>output,
>>>>>>>>>>>>> we fully intend that our release tarballs
would *not* have
>>>>>>>>>>>>>this
>>>>>>>>>>>>> requirement. Ie, when we go to cut a
release part of the
>>>>>>>>>>>>>process the
>>>>>>>>>>>>> RM would run would be to pull all of
those dependencies before
>>>>>>>>>>>>> creating a tarball that would be wholly
self contained. Given
>>>>>>>>>>>>>an
>>>>>>>>>>>>> apache-couchdb-x.y.z.tar.gz release file,
there won't be a
>>>>>>>>>>>>>requirement
>>>>>>>>>>>>> to have access to the ASF git repos.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not entirely sure how controversial
this is for anyone.
>>>>>>>>>>>>>For
>>>>>>>>>>>>>the
>>>>>>>>>>>>> most part the reactions I remember hearing
were more concerned
>>>>>>>>>>>>>on
>>>>>>>>>>>>> whether the infrastructure team would
allow us to use this
>>>>>>>>>>>>>sort
>>>>>>>>>>>>>of
>>>>>>>>>>>>> configuration. I looked yesterday and
asked and apparently its
>>>>>>>>>>>>> something we can request but as always
we'll want to verify
>>>>>>>>>>>>>again if
>>>>>>>>>>>>> we have consensus to move in this direction.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone have comments or flames? Right
now I'm just interested
>>>>>>>>>>>>>in
>>>>>>>>>>>>> feeling out what sort of (lack of?) consensus
there is on
>>>>>>>>>>>>>such a
>>>>>>>>>>>>> change. If there's general consensus
I'd think we'd do a vote
>>>>>>>>>>>>>in a
>>>>>>>>>>>>> couple weeks and if that passes then
start on down this road
>>>>>>>>>>>>>for the
>>>>>>>>>>>>> two merge projects and then it would
become part of master
>>>>>>>>>>>>>once
>>>>>>>>>>>>>those
>>>>>>>>>>>>> land (as opposed to doing this to master
and then attempting
>>>>>>>>>>>>>to
>>>>>>>>>>>>>merge
>>>>>>>>>>>>> rcouch/bigcouch onto that somehow).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is a quick pass at listing what
extra repositories I'd
>>>>>>>>>>>>>have
>>>>>>>>>>>>> created. Some of these applications only
exist in the bigcouch
>>>>>>>>>>>>>and/or
>>>>>>>>>>>>> rcouch branches so that's where the unfamiliar
application
>>>>>>>>>>>>>names are
>>>>>>>>>>>>> from. I'd also point out that the documentation
and fauxton
>>>>>>>>>>>>>things are
>>>>>>>>>>>>> just on a whim in that we could decouple
that development from
>>>>>>>>>>>>>the
>>>>>>>>>>>>> erlang development. I can see arguments
for an against those.
>>>>>>>>>>>>>I'm much
>>>>>>>>>>>>> less concerned on that aspect than the
Erlang parts that are
>>>>>>>>>>>>>directly
>>>>>>>>>>>>> affected by rebar/Erlang conventions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  chttpd
>>>>>>>>>>>>>  config
>>>>>>>>>>>>>  couch
>>>>>>>>>>>>>  couch_collate
>>>>>>>>>>>>>  couch_dbupdates
>>>>>>>>>>>>>  couch_httpd
>>>>>>>>>>>>>  couch_index
>>>>>>>>>>>>>  couch_mrview
>>>>>>>>>>>>>  couch_plugins
>>>>>>>>>>>>>  couch_replicator
>>>>>>>>>>>>>  documentation
>>>>>>>>>>>>>  ddoc_cache
>>>>>>>>>>>>>  ets_lru
>>>>>>>>>>>>>  fabric
>>>>>>>>>>>>>  fauxton
>>>>>>>>>>>>>  ibrowse
>>>>>>>>>>>>>  jiffy
>>>>>>>>>>>>>  mem3
>>>>>>>>>>>>>  mochiweb
>>>>>>>>>>>>>  oauth
>>>>>>>>>>>>>  rebar
>>>>>>>>>>>>>  rexi
>>>>>>>>>>>>>  snappy
>>>>>>>>>>>>>  twig
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I also contemplated this and and I am generally
+1 on this. And
>>>>>>>>>>>>definitely
>>>>>>>>>>>> +1 to mirror them on the apache git if possible.
 I have a
>>>>>>>>>>>>couple of
>>>>>>>>>>>> comments though.
>>>>>>>>>>>>
>>>>>>>>>>>> Initially I also had everything separated
in its own source
>>>>>>>>>>>>repository. 1
>>>>>>>>>>>> year ago I merged back as one core repo the
couchdb erlang
>>>>>>>>>>>>applications and
>>>>>>>>>>>> put all the dependencies in the refuge repository
or in the
>>>>>>>>>>>>refuge CDN for
>>>>>>>>>>>> the spidermonkey and ICU sources.
>>>>>>>>>>>>
>>>>>>>>>>>> I merged back as one core repo the couchdb
erlang applications
>>>>>>>>>>>>because they
>>>>>>>>>>>> were a little too much dependant. Especially
couch_httpd,
>>>>>>>>>>>>couch_index and
>>>>>>>>>>>> couch_mrview. These applications are not
yet enough by
>>>>>>>>>>>>themselves.
>>>>>>>>>>>>
>>>>>>>>>>>> Imo if we split everything in  their own
apps, then we should
>>>>>>>>>>>>make sure
>>>>>>>>>>>> that couch_httpd can be used without couch_index
and
>>>>>>>>>>>>couch_mrview (which
>>>>>>>>>>>> means that "all_docs" is available in couch_httpd).
Also we
>>>>>>>>>>>>should be able
>>>>>>>>>>>> to just launch couch without any of the above.
And probably
>>>>>>>>>>>>without the
>>>>>>>>>>>> need of an ini. The couch_query_server module
thing is an
>>>>>>>>>>>>interesting case.
>>>>>>>>>>>> bigcouch is also introducing `ddoc_cache`
which I am not sure
>>>>>>>>>>>>why it is
>>>>>>>>>>>> provided as a standalone app. Does it means
it can be replaced
>>>>>>>>>>>>by another
>>>>>>>>>>>> application eventually? Why not having it
simply in the  couch
>>>>>>>>>>>>application?
>>>>>>>>>>>> Does it needs to be updated separately?
>>>>>>>>>>>>
>>>>>>>>>>>> Also  all our base applications should also
be named spaced
>>>>>>>>>>>>correctly so
>>>>>>>>>>>> they will be strictly identified as erlang
modules:  "config"
>>>>>>>>>>>>is
>>>>>>>>>>>>too
>>>>>>>>>>>> generic, "ddoc_cache" too. Others are probably
OK.
>>>>>>>>>>>>
>>>>>>>>>>>> There are probably other things that we could
provide as apps:
>>>>>>>>>>>>
>>>>>>>>>>>> - couch_daemon,
>>>>>>>>>>>> - couch_js
>>>>>>>>>>>> - couch_external
>>>>>>>>>>>> - couch_stats
>>>>>>>>>>>> - couch_compaction_daemon
>>>>>>>>>>>> - couch_httpd_proxy
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway again i'm +1 for this move, I really
think it's a good
>>>>>>>>>>>>idea.
>>>>>>>>>>>>
>>>>>>>>>>>> - benoit
>>>>>>>>>>>
>>>>>>>>>>> I agree on most of this. Roughly I see three
general points.
>>>>>>>>>>>
>>>>>>>>>>> First, deciding on whether some things are external
deps is
>>>>>>>>>>>definitely up for discussion. Whether couch_mrview
is a different
>>>>>>>>>>>app/repo is not necessarily clear cut. Personally
I think I over
>>>>>>>>>>>engineered couch_index which blurs the lines a
bit. If I could
>>>>>>>>>>>wave a wand I'd have just couch_mrview and it'd
be separate. More
>>>>>>>>>>>importantly I think the separate repos makes these
things more
>>>>>>>>>>>apparent. The fact were discussing this sort of
architecture
>>>>>>>>>>>thing
>>>>>>>>>>>is suggestive that it's forcing us to think a
bit harder.
>>>>>>>>>>>
>>>>>>>>>>> Second is the aspect of composability. For instance
the mrview
>>>>>>>>>>>thing to me is obviously a different repo precisely
so a user
>>>>>>>>>>>could import couch (_core?) directly without requiring
the spider
>>>>>>>>>>>monkey dependency. The monolithic repo doesn't
allow this without
>>>>>>>>>>>some very non-standard tooling.
>>>>>>>>>>>
>>>>>>>>>>> Thirdly, app naming is always a contention. The
config name was
>>>>>>>>>>>actually a hot code upgrade concern. We couldn't
reuse
>>>>>>>>>>>couch_config directly at the time. And Adam was
also hopeful we
>>>>>>>>>>>could the it into a useful non-specific config
app.
>>>>>>>>>>>
>>>>>>>>>>> Fourthly, and related to secondly, we'll also
want to look at
>>>>>>>>>>>splitting other apps out as necessary. The ones
you listed I
>>>>>>>>>>>think
>>>>>>>>>>>aren't controversial it's just that no one has
done it yet. My
>>>>>>>>>>>list was purely what existed so far without attempting
to carve
>>>>>>>>>>>things up more. I definitely agree we should carve
more in just
>>>>>>>>>>>wanted to cover consensus that carving is the
right direction.
>>>>>>>>>>>
>>>>>>>>>>> Fifthly, I'm done typing on my phone. I'll fill
in more thoughts
>>>>>>>>>>>tomorrow.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Noah Slater
>>>>>> https://twitter.com/nslater
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Noah Slater
>>>>> https://twitter.com/nslater
>>>>
>>>
>>
>>
>>
>>--
>>Noah Slater
>>https://twitter.com/nslater
>



-- 
Noah Slater
https://twitter.com/nslater

Mime
View raw message