couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <>
Subject Re: Using rebar to install couchdb
Date Tue, 19 Oct 2010 19:04:32 GMT

On 18 Oct 2010, at 03:03, Paul Davis wrote:

> On Sun, Oct 17, 2010 at 3:20 PM, Jan Lehnardt <> wrote:
>> Ace, Paul :)
>> Just some bikeshedding as I agree with most that is said here.
>> I'm wondering if the OS process handling could be its own module or
>> part of couch_core. While the View Server is the primary user, it is
>> not the only one and having OS process management in the view server
>> module just feels wrong to me :)
>> Cheers
>> Jan
>> --
> Damn. I was kinda hoping no one would notice that until after I had an
> answer. Thanks for ruining my plan.

Sorry :)

I think your plan of doing the separation separately from rearchitecing
the source is solid.


> I've been thinking over it for the last few days and my current
> favorite idea is mostly similar to what you suggest in making an app
> that does generic OS process handling like we're discussing in the
> thread about refactoring couch_query_servers.erl. The issue at hand is
> figuring out a method to make this mix in well with our current
> situation of reusing those OS processes for  multiple duties. My
> general thinking in this direction is that this API will end up
> evolving a bit to handle the ability to be extended as we add new OS
> level functionality.
> My primary goal for the source tree layout is to change as little
> source code as possible so I was planning on delaying internal
> refactoring until after we land the retooling of the build mechanics
> so that our svn history retains some semblance of sanity as we move
> through this.
> My current plan of attack is to hack a shell script and a couple small
> diffs to make the source move. Once I get that project up and running
> I'll start a JIRA ticket so we have a place for focusing detail
> oriented discussions.
> Paul Davis
>> On 14 Oct 2010, at 20:54, Paul Davis wrote:
>>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <>
>>>> In an attempt to start some merging with cloudant I would like to
>>>> start by using rebar in our install process.
>>>> Like i see it, we could continue to use autotools to create the
>>>> rebar.config files and other templates an then rebar for the final
>>>> build and dependencies management. This changes as noticed by @davisp
>>>> also imply we make our tree a little more OTP compliant. I would like
>>>> to start this work asap.
>>>> Thoughts ?
>>>> - benoit
>>> So there's a couple issues at hand here which seem to be motivated by
>>> the desire to start using tools like rebar.
>>> Our current source tree is not compliant with some of the basic
>>> Erlang/OTP conventions. This is both bad technically and socially.
>>> Technically, it prevents us from easily integrating tools like rebar
>>> that would help advanced users with things like making Erlang reltools
>>> packages. Socially, it doesn't reflect well on us to members of the
>>> Erlang community that may have otherwise become contributors. All
>>> languages have a standard package layout and Erlang is no different.
>>> The current CouchDB Erlang app has grown considerably. There's been
>>> general consensus that we need to start splitting it up into smaller
>>> applications that encompass specific functionality. There's been a bit
>>> of effort in this direction, but its such a major change to source
>>> file location it needs to have a community consensus to really start
>>> working on seriously.
>>> I don't think we should focus directly on the issue of integrating
>>> rebar. It should definitely be a goal, but not at the cost of our
>>> current situation. Noah Slater has maintained an excellent build
>>> system for us as is shown by the number of people building CouchDB
>>> from source and the number of packages available. While I have argued
>>> with him on numerous occasions about details, I have come to the
>>> conclusion that it is not possible for him to be wrong. I personally
>>> attribute this to the fact that he's most likely an advanced robot
>>> from the future. That said, Noah has voiced concerns to various ideas
>>> and we should make sure that any of his concerns are fully addressed.
>>> We should attempt to make sure that any tool support doesn't morph
>>> into tool requirement. For instance, I think we should make sure that
>>> its possible to keep compiling CouchDB without rebar and not come to
>>> rely on it.
>>> While I'd be more than happy to start in on this and handle all of the
>>> build system refactoring to make this happen, I'm not going to start
>>> until there's a community consensus on what needs to be done. There
>>> are a couple paths that I could see us taking to make this happen. We
>>> could just make the current source tree be rebar compatible and figure
>>> out the build system to do the optional rebar build or we could also
>>> take this chance to split the source code into multiple applications.
>>> Personally, I'd prefer to take this opportunity to organize the code
>>> with multiple erlang apps.
>>> Too get the conversation rolling here's a first pass at a new app proposal:
>>> etap:
>>>    Nick Gerakines now releases etap as a single .erl file that can be
>>> dropped into the test directory. This app should be removed in favor
>>> of that method.
>>> erlang-oauth:
>>>    Should be renamed to just oauth. That erlang- prefix has bugged me
>>> fore entirely too long.
>>> mochiweb, ibrowse, oauth:
>>>    Refactored to use standard src, include, ebin, priv directories to
>>> be OTP compliant. This results in directories like
>>>        ./src/$APP/ebin
>>>        ./src/$APP/incldue
>>>        ./src/$APP/priv
>>>        ./src/$APP/src
>>> couchdb:
>>>    Each proposed app will be structured as described above. Proposed apps:
>>>    couch_core: The core Erlang modules for storing docs and managing
>>> "internal infrastructure"
>>>    couch_view: The view engine as well as the holder for managing OS processes.
>>>    couch_rep: couch_rep*.erl
>>>    couch_externals: couch_external*.erl
>>>    couch_httpd: couch_http*.erl
>>> At the bottom of this email I made an initial pass through the
>>> ./src/couchdb tree to classify file by file into the described apps.
>>> There are also some minor warts in this split. Things like the core
>>> couchdb classes currently require calling into the couch_view app and
>>> what not. Given the "interesting" call graph for CouchDb internals, I
>>> think we should avoid addressing this and aim to minimize the actual
>>> change to source file contents during this move. After modules have
>>> been moved we can then start on teasing out the various cyclic
>>> dependencies as necessary.
>>> Also, on a last note, seeing as this is a major change to the source
>>> tree (and ideally 0 change to source code) it might be a good idea to
>>> start this work just after we release 1.1.0.
>>> What do you guys think?
>>> Paul Davis
>>> List of files by per app:
>>> couch_core:
>>>    ./src/couchdb/couch.erl
>>>    ./src/couchdb/couch_app.erl
>>>    ./src/couchdb/couch_auth_cache.erl
>>>    ./src/couchdb/couch_btree.erl
>>>    ./src/couchdb/couch_changes.erl
>>>    ./src/couchdb/couch_config.erl
>>>    ./src/couchdb/couch_config_writer.erl
>>>    ./src/couchdb/couch_db.erl
>>>    ./src/couchdb/couch_db.hrl
>>>    ./src/couchdb/couch_db_update_notifier.erl
>>>    ./src/couchdb/couch_db_update_notifier_sup.erl
>>>    ./src/couchdb/couch_db_updater.erl
>>>    ./src/couchdb/couch_doc.erl
>>>    ./src/couchdb/couch_event_sup.erl
>>>    ./src/couchdb/couch_file.erl
>>>    ./src/couchdb/couch_key_tree.erl
>>>    ./src/couchdb/couch_log.erl
>>>    ./src/couchdb/couch_ref_counter.erl
>>>    ./src/couchdb/couch_server.erl
>>>    ./src/couchdb/couch_server_sup.erl
>>>    ./src/couchdb/couch_stats_aggregator.erl
>>>    ./src/couchdb/couch_stats_collector.erl
>>>    ./src/couchdb/couch_stream.erl
>>>    ./src/couchdb/couch_task_status.erl
>>>    ./src/couchdb/couch_util.erl
>>>    ./src/couchdb/couch_uuids.erl
>>>    ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>>    ./src/couchdb/priv/
>>> couch_view:
>>>    ./src/couchdb/couch_native_process.erl
>>>    ./src/couchdb/couch_os_process.erl
>>>    ./src/couchdb/couch_query_servers.erl
>>>    ./src/couchdb/couch_view.erl
>>>    ./src/couchdb/couch_view_compactor.erl
>>>    ./src/couchdb/couch_view_group.erl
>>>    ./src/couchdb/couch_view_updater.erl
>>>    ./src/couchdb/couch_work_queue.erl
>>>    ./src/couchdb/priv/couch_js/http.c
>>>    ./src/couchdb/priv/couch_js/http.h
>>>    ./src/couchdb/priv/couch_js/main.c
>>>    ./src/couchdb/priv/couch_js/utf8.c
>>>    ./src/couchdb/priv/couch_js/utf8.h
>>>    ./src/couchdb/priv/spawnkillable/
>>>    ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>>> couch_rep:
>>>    ./src/couchdb/couch_js_functions.hrl
>>>    ./src/couchdb/couch_rep.erl
>>>    ./src/couchdb/couch_rep_att.erl
>>>    ./src/couchdb/couch_rep_changes_feed.erl
>>>    ./src/couchdb/couch_rep_db_listener.erl
>>>    ./src/couchdb/couch_rep_httpc.erl
>>>    ./src/couchdb/couch_rep_missing_revs.erl
>>>    ./src/couchdb/couch_rep_reader.erl
>>>    ./src/couchdb/couch_rep_sup.erl
>>>    ./src/couchdb/couch_rep_writer.erl
>>> couch_externals:
>>>    ./src/couchdb/couch_external_manager.erl
>>>    ./src/couchdb/couch_external_server.erl
>>> couch_httpd:
>>>    ./src/couchdb/couch_httpd.erl
>>>    ./src/couchdb/couch_httpd_auth.erl
>>>    ./src/couchdb/couch_httpd_db.erl
>>>    ./src/couchdb/couch_httpd_external.erl
>>>    ./src/couchdb/couch_httpd_misc_handlers.erl
>>>    ./src/couchdb/couch_httpd_oauth.erl
>>>    ./src/couchdb/couch_httpd_rewrite.erl
>>>    ./src/couchdb/couch_httpd_show.erl
>>>    ./src/couchdb/couch_httpd_stats_handlers.erl
>>>    ./src/couchdb/couch_httpd_vhost.erl
>>>    ./src/couchdb/couch_httpd_view.erl

View raw message