Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F1C0810750 for ; Fri, 17 Jan 2014 21:36:36 +0000 (UTC) Received: (qmail 53793 invoked by uid 500); 17 Jan 2014 21:36:36 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 53740 invoked by uid 500); 17 Jan 2014 21:36:36 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 53730 invoked by uid 99); 17 Jan 2014 21:36:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jan 2014 21:36:35 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [72.246.2.14] (HELO prod-mail-xrelay02.akamai.com) (72.246.2.14) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jan 2014 21:36:32 +0000 Received: from prod-mail-xrelay02.akamai.com (localhost [127.0.0.1]) by postfix.imss70 (Postfix) with ESMTP id D3B2C2853B for ; Fri, 17 Jan 2014 21:36:10 +0000 (GMT) Received: from prod-mail-relay06.akamai.com (prod-mail-relay06.akamai.com [172.17.120.126]) by prod-mail-xrelay02.akamai.com (Postfix) with ESMTP id C2A9A2852A for ; Fri, 17 Jan 2014 21:36:10 +0000 (GMT) Received: from ustx2ex-cashub.dfw01.corp.akamai.com (ustx2ex-cashub1.dfw01.corp.akamai.com [172.27.25.75]) by prod-mail-relay06.akamai.com (Postfix) with ESMTP id 94DBF2029 for ; Fri, 17 Jan 2014 21:36:10 +0000 (GMT) Received: from USMBX2.msg.corp.akamai.com ([169.254.1.225]) by ustx2ex-cashub1.dfw01.corp.akamai.com ([172.27.25.75]) with mapi; Fri, 17 Jan 2014 15:36:08 -0600 From: "Mutton, James" To: "dev@couchdb.apache.org" Date: Fri, 17 Jan 2014 15:36:05 -0600 Subject: Re: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies Thread-Topic: [DISCUSS] Multiple Repositories for Erlang Apps and Dependencies Thread-Index: Ac8TzCH6hf7OnuFWQsCJbuDOPIyJMQ== Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.0.101115 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Excellent! Done, thanks. On 1/17/14 11:44 AM, "Noah Slater" wrote: >Done. Edit away! (Thanks!) > >On 17 January 2014 20:28, Mutton, James wrote: >> JamesMutton >> >> >> >> >> >> >> On 1/17/14 11:09 AM, "Noah Slater" wrote: >> >>>Good plan. What's your wiki username? I'll add you. >>> >>>On 17 January 2014 20:01, Mutton, James wrote: >>>> Might I suggest that this sounds like good info to document on the >>>>wiki >>>> for committers getting started. I'd add it but I'm not in the >>>>allow-list. >>>> :) >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 1/17/14 4:00 AM, "Garren Smith" wrote: >>>> >>>>>I'm claiming 2nd person added! >>>>> >>>>>On 17 Jan 2014, at 1:28 PM, Noah Slater wrote: >>>>> >>>>>> Psst. A little birdy tells me that if you ask nicely, the infra >>>>>>folks >>>>>> will add you to the Apache GitHub org too, so you can show off your >>>>>> Apache affiliation. I was the first person added. Because I may have >>>>>> been the first to ask. ;) >>>>>> >>>>>> On 17 January 2014 11:56, Noah Slater wrote: >>>>>>> Awesome, thanks Paul. >>>>>>> >>>>>>> Note to all devs: if you want your contributions to CouchDB to show >>>>>>>up >>>>>>> on your GitHub profile, you have to star each of the repositories. >>>>>>> (That's just how GitHub mechanics work for repo mirrors.) >>>>>>> >>>>>>> You can find them all here: >>>>>>> >>>>>>> https://github.com/apache >>>>>>> >>>>>>> On 17 January 2014 00:00, Paul Davis >>>>>>>wrote: >>>>>>>> New repos are up: >>>>>>>>https://git-wip-us.apache.org/repos/asf?s=3Dcouchdb >>>>>>>> >>>>>>>> I'm gonna go through and initialize them with history from master >>>>>>>>or >>>>>>>> one of the bigcouch and rcouch branches as appropriate. >>>>>>>> >>>>>>>> On Thu, Jan 16, 2014 at 2:12 PM, Paul Davis >>>>>>>> wrote: >>>>>>>>> Infrastructure ticket opened: >>>>>>>>>https://issues.apache.org/jira/browse/INFRA-7203 >>>>>>>>> >>>>>>>>> On Thu, Jan 16, 2014 at 1:42 PM, Jan Lehnardt >>>>>>>>>wrote: >>>>>>>>>> >>>>>>>>>> On 16 Jan 2014, at 20:42 , Paul Davis >>>>>>>>>> >>>>>>>>>>wrote: >>>>>>>>>> >>>>>>>>>>> It doesn't appear that this is objectionable to anyone. Does >>>>>>>>>>>anyone >>>>>>>>>>> have an objection to us having infra/me create these repos to >>>>>>>>>>>use >>>>>>>>>>>for >>>>>>>>>>> the bigcouch/rcouch merge work? This won't affect master or >>>>>>>>>>>releases >>>>>>>>>>> until those merges finish. >>>>>>>>>> >>>>>>>>>> no objections. >>>>>>>>>> >>>>>>>>>> Jan >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 14, 2014 at 11:02 PM, Paul J Davis >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Jan 14, 2014, at 8:37 PM, Benoit Chesneau >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 15, 2014 at 12:22 AM, Paul Davis >>>>>>>>>>>>>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I've recently been having discussions about how to handle >>>>>>>>>>>>>>the >>>>>>>>>>>>>> repository configuration for various bits of CouchDB >>>>>>>>>>>>>>post-merge. The >>>>>>>>>>>>>> work that Benoit has been doing on the rcouch merge branch >>>>>>>>>>>>>>have >>>>>>>>>>>>>>also >>>>>>>>>>>>>> touched on this topic as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The background for those unfamiliar is that the standard >>>>>>>>>>>>>>operating >>>>>>>>>>>>>> procedure for Erlang is to have a single Erlang application >>>>>>>>>>>>>>per >>>>>>>>>>>>>> repository and then rely on rebar to fetch each dependency. >>>>>>>>>>>>>> Traditionally in CouchDB land we've always just included the >>>>>>>>>>>>>>source to >>>>>>>>>>>>>> all applications in a single monolithic repository and >>>>>>>>>>>>>>periodically >>>>>>>>>>>>>> reimport changes from upstream dependencies. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Recently rcouch changed from the monolithic repository to >>>>>>>>>>>>>>use >>>>>>>>>>>>>>external >>>>>>>>>>>>>> repositories for some dependencies. Originally the BigCouch >>>>>>>>>>>>>>used an >>>>>>>>>>>>>> even more federated scheme that had each Erlang application >>>>>>>>>>>>>>in >>>>>>>>>>>>>>an >>>>>>>>>>>>>> external repository (and the core couch Erlang application >>>>>>>>>>>>>>was >>>>>>>>>>>>>>in the >>>>>>>>>>>>>> root repository). When Bob Newson and I did the initial >>>>>>>>>>>>>>hacking >>>>>>>>>>>>>>on the >>>>>>>>>>>>>> BigCouch merge we pulled those external dependencies into >>>>>>>>>>>>>>the >>>>>>>>>>>>>>root >>>>>>>>>>>>>> repository reverting back to the large monolithic approach. >>>>>>>>>>>>>> >>>>>>>>>>>>>> After trying to deal with the merge and contemplating how >>>>>>>>>>>>>>various >>>>>>>>>>>>>> Erlang release things might work it's become fairly apparent >>>>>>>>>>>>>>that the >>>>>>>>>>>>>> monolithic approach is a bit constrictive. For instance, >>>>>>>>>>>>>>part >>>>>>>>>>>>>>of >>>>>>>>>>>>>> rebar's versioning abilities lets you tag repositories to >>>>>>>>>>>>>>generate >>>>>>>>>>>>>> versions rather than manually updating versions in source >>>>>>>>>>>>>>files. >>>>>>>>>>>>>> Another thing I've found on other projects is that having >>>>>>>>>>>>>>each >>>>>>>>>>>>>> application in a separate repository requires developers to >>>>>>>>>>>>>>think a >>>>>>>>>>>>>> bit more detailed about the public internal interfaces used >>>>>>>>>>>>>>through >>>>>>>>>>>>>> out the system. We've done some work to this extent already >>>>>>>>>>>>>>with >>>>>>>>>>>>>> separating source directories but forcing commits to >>>>>>>>>>>>>>multiple >>>>>>>>>>>>>> repositories shoots up a big red flag that maybe there's a >>>>>>>>>>>>>>high >>>>>>>>>>>>>>level >>>>>>>>>>>>>> of coupling between two bits of code. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Other benefits of having the multiple repository setup is >>>>>>>>>>>>>>that >>>>>>>>>>>>>>its >>>>>>>>>>>>>> possible that this lends itself to being integrated with the >>>>>>>>>>>>>>proposed >>>>>>>>>>>>>> plugin system. It'd be fairly trivial to have a script that >>>>>>>>>>>>>>went and >>>>>>>>>>>>>> fetched plugins that aren't developed at Apache (as a >>>>>>>>>>>>>>./configure time >>>>>>>>>>>>>> switch type of thing). Having a system like this would also >>>>>>>>>>>>>>allow us >>>>>>>>>>>>>> to have groups focused on particular bits of development not >>>>>>>>>>>>>>have to >>>>>>>>>>>>>> concern themselves with the unrelated parts of the system. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Given all that, I'd like to propose that we move to having a >>>>>>>>>>>>>> repository for each application/dependency that we use to >>>>>>>>>>>>>>build >>>>>>>>>>>>>> CouchDB. Each repository would be hosted on ASF infra and >>>>>>>>>>>>>>mirrored to >>>>>>>>>>>>>> GitHub as expected. This means that we could have the root >>>>>>>>>>>>>>repository >>>>>>>>>>>>>> be a simple repo that contains packaging/release/build stuff >>>>>>>>>>>>>>that >>>>>>>>>>>>>> would enable lots of the ideas offered on configurable types >>>>>>>>>>>>>>of >>>>>>>>>>>>>> release generation. I've included an initial list of >>>>>>>>>>>>>>repositories at >>>>>>>>>>>>>> the end of this email. Its basically just the apps that have >>>>>>>>>>>>>>been >>>>>>>>>>>>>> split out in either rcouch or bigcouch plus a few other bits >>>>>>>>>>>>>>from >>>>>>>>>>>>>> CouchDB master. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would also point out that even though our main repo would >>>>>>>>>>>>>>need to >>>>>>>>>>>>>> fetch other dependencies from the internet to build the >>>>>>>>>>>>>>final >>>>>>>>>>>>>>output, >>>>>>>>>>>>>> we fully intend that our release tarballs would *not* have >>>>>>>>>>>>>>this >>>>>>>>>>>>>> requirement. Ie, when we go to cut a release part of the >>>>>>>>>>>>>>process the >>>>>>>>>>>>>> RM would run would be to pull all of those dependencies >>>>>>>>>>>>>>before >>>>>>>>>>>>>> creating a tarball that would be wholly self contained. >>>>>>>>>>>>>>Given >>>>>>>>>>>>>>an >>>>>>>>>>>>>> apache-couchdb-x.y.z.tar.gz release file, there won't be a >>>>>>>>>>>>>>requirement >>>>>>>>>>>>>> to have access to the ASF git repos. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm not entirely sure how controversial this is for anyone. >>>>>>>>>>>>>>For >>>>>>>>>>>>>>the >>>>>>>>>>>>>> most part the reactions I remember hearing were more >>>>>>>>>>>>>>concerned >>>>>>>>>>>>>>on >>>>>>>>>>>>>> whether the infrastructure team would allow us to use this >>>>>>>>>>>>>>sort >>>>>>>>>>>>>>of >>>>>>>>>>>>>> configuration. I looked yesterday and asked and apparently >>>>>>>>>>>>>>its >>>>>>>>>>>>>> something we can request but as always we'll want to verify >>>>>>>>>>>>>>again if >>>>>>>>>>>>>> we have consensus to move in this direction. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyone have comments or flames? Right now I'm just >>>>>>>>>>>>>>interested >>>>>>>>>>>>>>in >>>>>>>>>>>>>> feeling out what sort of (lack of?) consensus there is on >>>>>>>>>>>>>>such a >>>>>>>>>>>>>> change. If there's general consensus I'd think we'd do a >>>>>>>>>>>>>>vote >>>>>>>>>>>>>>in a >>>>>>>>>>>>>> couple weeks and if that passes then start on down this road >>>>>>>>>>>>>>for the >>>>>>>>>>>>>> two merge projects and then it would become part of master >>>>>>>>>>>>>>once >>>>>>>>>>>>>>those >>>>>>>>>>>>>> land (as opposed to doing this to master and then attempting >>>>>>>>>>>>>>to >>>>>>>>>>>>>>merge >>>>>>>>>>>>>> rcouch/bigcouch onto that somehow). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is a quick pass at listing what extra repositories I'd >>>>>>>>>>>>>>have >>>>>>>>>>>>>> created. Some of these applications only exist in the >>>>>>>>>>>>>>bigcouch >>>>>>>>>>>>>>and/or >>>>>>>>>>>>>> rcouch branches so that's where the unfamiliar application >>>>>>>>>>>>>>names are >>>>>>>>>>>>>> from. I'd also point out that the documentation and fauxton >>>>>>>>>>>>>>things are >>>>>>>>>>>>>> just on a whim in that we could decouple that development >>>>>>>>>>>>>>from >>>>>>>>>>>>>>the >>>>>>>>>>>>>> erlang development. I can see arguments for an against >>>>>>>>>>>>>>those. >>>>>>>>>>>>>>I'm much >>>>>>>>>>>>>> less concerned on that aspect than the Erlang parts that are >>>>>>>>>>>>>>directly >>>>>>>>>>>>>> affected by rebar/Erlang conventions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> chttpd >>>>>>>>>>>>>> config >>>>>>>>>>>>>> couch >>>>>>>>>>>>>> couch_collate >>>>>>>>>>>>>> couch_dbupdates >>>>>>>>>>>>>> couch_httpd >>>>>>>>>>>>>> couch_index >>>>>>>>>>>>>> couch_mrview >>>>>>>>>>>>>> couch_plugins >>>>>>>>>>>>>> couch_replicator >>>>>>>>>>>>>> documentation >>>>>>>>>>>>>> ddoc_cache >>>>>>>>>>>>>> ets_lru >>>>>>>>>>>>>> fabric >>>>>>>>>>>>>> fauxton >>>>>>>>>>>>>> ibrowse >>>>>>>>>>>>>> jiffy >>>>>>>>>>>>>> mem3 >>>>>>>>>>>>>> mochiweb >>>>>>>>>>>>>> oauth >>>>>>>>>>>>>> rebar >>>>>>>>>>>>>> rexi >>>>>>>>>>>>>> snappy >>>>>>>>>>>>>> twig >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I also contemplated this and and I am generally +1 on this. >>>>>>>>>>>>>And >>>>>>>>>>>>>definitely >>>>>>>>>>>>> +1 to mirror them on the apache git if possible. I have a >>>>>>>>>>>>>couple of >>>>>>>>>>>>> comments though. >>>>>>>>>>>>> >>>>>>>>>>>>> Initially I also had everything separated in its own source >>>>>>>>>>>>>repository. 1 >>>>>>>>>>>>> year ago I merged back as one core repo the couchdb erlang >>>>>>>>>>>>>applications and >>>>>>>>>>>>> put all the dependencies in the refuge repository or in the >>>>>>>>>>>>>refuge CDN for >>>>>>>>>>>>> the spidermonkey and ICU sources. >>>>>>>>>>>>> >>>>>>>>>>>>> I merged back as one core repo the couchdb erlang >>>>>>>>>>>>>applications >>>>>>>>>>>>>because they >>>>>>>>>>>>> were a little too much dependant. Especially couch_httpd, >>>>>>>>>>>>>couch_index and >>>>>>>>>>>>> couch_mrview. These applications are not yet enough by >>>>>>>>>>>>>themselves. >>>>>>>>>>>>> >>>>>>>>>>>>> Imo if we split everything in their own apps, then we should >>>>>>>>>>>>>make sure >>>>>>>>>>>>> that couch_httpd can be used without couch_index and >>>>>>>>>>>>>couch_mrview (which >>>>>>>>>>>>> means that "all_docs" is available in couch_httpd). Also we >>>>>>>>>>>>>should be able >>>>>>>>>>>>> to just launch couch without any of the above. And probably >>>>>>>>>>>>>without the >>>>>>>>>>>>> need of an ini. The couch_query_server module thing is an >>>>>>>>>>>>>interesting case. >>>>>>>>>>>>> bigcouch is also introducing `ddoc_cache` which I am not sure >>>>>>>>>>>>>why it is >>>>>>>>>>>>> provided as a standalone app. Does it means it can be >>>>>>>>>>>>>replaced >>>>>>>>>>>>>by another >>>>>>>>>>>>> application eventually? Why not having it simply in the >>>>>>>>>>>>>couch >>>>>>>>>>>>>application? >>>>>>>>>>>>> Does it needs to be updated separately? >>>>>>>>>>>>> >>>>>>>>>>>>> Also all our base applications should also be named spaced >>>>>>>>>>>>>correctly so >>>>>>>>>>>>> they will be strictly identified as erlang modules: "config" >>>>>>>>>>>>>is >>>>>>>>>>>>>too >>>>>>>>>>>>> generic, "ddoc_cache" too. Others are probably OK. >>>>>>>>>>>>> >>>>>>>>>>>>> There are probably other things that we could provide as >>>>>>>>>>>>>apps: >>>>>>>>>>>>> >>>>>>>>>>>>> - couch_daemon, >>>>>>>>>>>>> - couch_js >>>>>>>>>>>>> - couch_external >>>>>>>>>>>>> - couch_stats >>>>>>>>>>>>> - couch_compaction_daemon >>>>>>>>>>>>> - couch_httpd_proxy >>>>>>>>>>>>> >>>>>>>>>>>>> Anyway again i'm +1 for this move, I really think it's a good >>>>>>>>>>>>>idea. >>>>>>>>>>>>> >>>>>>>>>>>>> - benoit >>>>>>>>>>>> >>>>>>>>>>>> I agree on most of this. Roughly I see three general points. >>>>>>>>>>>> >>>>>>>>>>>> First, deciding on whether some things are external deps is >>>>>>>>>>>>definitely up for discussion. Whether couch_mrview is a >>>>>>>>>>>>different >>>>>>>>>>>>app/repo is not necessarily clear cut. Personally I think I >>>>>>>>>>>>over >>>>>>>>>>>>engineered couch_index which blurs the lines a bit. If I could >>>>>>>>>>>>wave a wand I'd have just couch_mrview and it'd be separate. >>>>>>>>>>>>More >>>>>>>>>>>>importantly I think the separate repos makes these things more >>>>>>>>>>>>apparent. The fact were discussing this sort of architecture >>>>>>>>>>>>thing >>>>>>>>>>>>is suggestive that it's forcing us to think a bit harder. >>>>>>>>>>>> >>>>>>>>>>>> Second is the aspect of composability. For instance the mrview >>>>>>>>>>>>thing to me is obviously a different repo precisely so a user >>>>>>>>>>>>could import couch (_core?) directly without requiring the >>>>>>>>>>>>spider >>>>>>>>>>>>monkey dependency. The monolithic repo doesn't allow this >>>>>>>>>>>>without >>>>>>>>>>>>some very non-standard tooling. >>>>>>>>>>>> >>>>>>>>>>>> Thirdly, app naming is always a contention. The config name >>>>>>>>>>>>was >>>>>>>>>>>>actually a hot code upgrade concern. We couldn't reuse >>>>>>>>>>>>couch_config directly at the time. And Adam was also hopeful we >>>>>>>>>>>>could the it into a useful non-specific config app. >>>>>>>>>>>> >>>>>>>>>>>> Fourthly, and related to secondly, we'll also want to look at >>>>>>>>>>>>splitting other apps out as necessary. The ones you listed I >>>>>>>>>>>>think >>>>>>>>>>>>aren't controversial it's just that no one has done it yet. My >>>>>>>>>>>>list was purely what existed so far without attempting to carve >>>>>>>>>>>>things up more. I definitely agree we should carve more in just >>>>>>>>>>>>wanted to cover consensus that carving is the right direction. >>>>>>>>>>>> >>>>>>>>>>>> Fifthly, I'm done typing on my phone. I'll fill in more >>>>>>>>>>>>thoughts >>>>>>>>>>>>tomorrow. >>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Noah Slater >>>>>>> https://twitter.com/nslater >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Noah Slater >>>>>> https://twitter.com/nslater >>>>> >>>> >>> >>> >>> >>>-- >>>Noah Slater >>>https://twitter.com/nslater >> > > > >-- >Noah Slater >https://twitter.com/nslater