couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <>
Subject Re: Using rebar to install couchdb
Date Mon, 18 Oct 2010 01:03:13 GMT
On Sun, Oct 17, 2010 at 3:20 PM, Jan Lehnardt <> wrote:
> Ace, Paul :)
> Just some bikeshedding as I agree with most that is said here.
> I'm wondering if the OS process handling could be its own module or
> part of couch_core. While the View Server is the primary user, it is
> not the only one and having OS process management in the view server
> module just feels wrong to me :)
> Cheers
> Jan
> --

Damn. I was kinda hoping no one would notice that until after I had an
answer. Thanks for ruining my plan.

I've been thinking over it for the last few days and my current
favorite idea is mostly similar to what you suggest in making an app
that does generic OS process handling like we're discussing in the
thread about refactoring couch_query_servers.erl. The issue at hand is
figuring out a method to make this mix in well with our current
situation of reusing those OS processes for  multiple duties. My
general thinking in this direction is that this API will end up
evolving a bit to handle the ability to be extended as we add new OS
level functionality.

My primary goal for the source tree layout is to change as little
source code as possible so I was planning on delaying internal
refactoring until after we land the retooling of the build mechanics
so that our svn history retains some semblance of sanity as we move
through this.

My current plan of attack is to hack a shell script and a couple small
diffs to make the source move. Once I get that project up and running
I'll start a JIRA ticket so we have a place for focusing detail
oriented discussions.

Paul Davis

> On 14 Oct 2010, at 20:54, Paul Davis wrote:
>> On Wed, Oct 13, 2010 at 5:23 PM, Benoit Chesneau <> wrote:
>>> In an attempt to start some merging with cloudant I would like to
>>> start by using rebar in our install process.
>>> Like i see it, we could continue to use autotools to create the
>>> rebar.config files and other templates an then rebar for the final
>>> build and dependencies management. This changes as noticed by @davisp
>>> also imply we make our tree a little more OTP compliant. I would like
>>> to start this work asap.
>>> Thoughts ?
>>> - benoit
>> So there's a couple issues at hand here which seem to be motivated by
>> the desire to start using tools like rebar.
>> Our current source tree is not compliant with some of the basic
>> Erlang/OTP conventions. This is both bad technically and socially.
>> Technically, it prevents us from easily integrating tools like rebar
>> that would help advanced users with things like making Erlang reltools
>> packages. Socially, it doesn't reflect well on us to members of the
>> Erlang community that may have otherwise become contributors. All
>> languages have a standard package layout and Erlang is no different.
>> The current CouchDB Erlang app has grown considerably. There's been
>> general consensus that we need to start splitting it up into smaller
>> applications that encompass specific functionality. There's been a bit
>> of effort in this direction, but its such a major change to source
>> file location it needs to have a community consensus to really start
>> working on seriously.
>> I don't think we should focus directly on the issue of integrating
>> rebar. It should definitely be a goal, but not at the cost of our
>> current situation. Noah Slater has maintained an excellent build
>> system for us as is shown by the number of people building CouchDB
>> from source and the number of packages available. While I have argued
>> with him on numerous occasions about details, I have come to the
>> conclusion that it is not possible for him to be wrong. I personally
>> attribute this to the fact that he's most likely an advanced robot
>> from the future. That said, Noah has voiced concerns to various ideas
>> and we should make sure that any of his concerns are fully addressed.
>> We should attempt to make sure that any tool support doesn't morph
>> into tool requirement. For instance, I think we should make sure that
>> its possible to keep compiling CouchDB without rebar and not come to
>> rely on it.
>> While I'd be more than happy to start in on this and handle all of the
>> build system refactoring to make this happen, I'm not going to start
>> until there's a community consensus on what needs to be done. There
>> are a couple paths that I could see us taking to make this happen. We
>> could just make the current source tree be rebar compatible and figure
>> out the build system to do the optional rebar build or we could also
>> take this chance to split the source code into multiple applications.
>> Personally, I'd prefer to take this opportunity to organize the code
>> with multiple erlang apps.
>> Too get the conversation rolling here's a first pass at a new app proposal:
>> etap:
>>    Nick Gerakines now releases etap as a single .erl file that can be
>> dropped into the test directory. This app should be removed in favor
>> of that method.
>> erlang-oauth:
>>    Should be renamed to just oauth. That erlang- prefix has bugged me
>> fore entirely too long.
>> mochiweb, ibrowse, oauth:
>>    Refactored to use standard src, include, ebin, priv directories to
>> be OTP compliant. This results in directories like
>>        ./src/$APP/ebin
>>        ./src/$APP/incldue
>>        ./src/$APP/priv
>>        ./src/$APP/src
>> couchdb:
>>    Each proposed app will be structured as described above. Proposed apps:
>>    couch_core: The core Erlang modules for storing docs and managing
>> "internal infrastructure"
>>    couch_view: The view engine as well as the holder for managing OS processes.
>>    couch_rep: couch_rep*.erl
>>    couch_externals: couch_external*.erl
>>    couch_httpd: couch_http*.erl
>> At the bottom of this email I made an initial pass through the
>> ./src/couchdb tree to classify file by file into the described apps.
>> There are also some minor warts in this split. Things like the core
>> couchdb classes currently require calling into the couch_view app and
>> what not. Given the "interesting" call graph for CouchDb internals, I
>> think we should avoid addressing this and aim to minimize the actual
>> change to source file contents during this move. After modules have
>> been moved we can then start on teasing out the various cyclic
>> dependencies as necessary.
>> Also, on a last note, seeing as this is a major change to the source
>> tree (and ideally 0 change to source code) it might be a good idea to
>> start this work just after we release 1.1.0.
>> What do you guys think?
>> Paul Davis
>> List of files by per app:
>> couch_core:
>>    ./src/couchdb/couch.erl
>>    ./src/couchdb/couch_app.erl
>>    ./src/couchdb/couch_auth_cache.erl
>>    ./src/couchdb/couch_btree.erl
>>    ./src/couchdb/couch_changes.erl
>>    ./src/couchdb/couch_config.erl
>>    ./src/couchdb/couch_config_writer.erl
>>    ./src/couchdb/couch_db.erl
>>    ./src/couchdb/couch_db.hrl
>>    ./src/couchdb/couch_db_update_notifier.erl
>>    ./src/couchdb/couch_db_update_notifier_sup.erl
>>    ./src/couchdb/couch_db_updater.erl
>>    ./src/couchdb/couch_doc.erl
>>    ./src/couchdb/couch_event_sup.erl
>>    ./src/couchdb/couch_file.erl
>>    ./src/couchdb/couch_key_tree.erl
>>    ./src/couchdb/couch_log.erl
>>    ./src/couchdb/couch_ref_counter.erl
>>    ./src/couchdb/couch_server.erl
>>    ./src/couchdb/couch_server_sup.erl
>>    ./src/couchdb/couch_stats_aggregator.erl
>>    ./src/couchdb/couch_stats_collector.erl
>>    ./src/couchdb/couch_stream.erl
>>    ./src/couchdb/couch_task_status.erl
>>    ./src/couchdb/couch_util.erl
>>    ./src/couchdb/couch_uuids.erl
>>    ./src/couchdb/priv/icu_driver/couch_icu_driver.c
>>    ./src/couchdb/priv/
>> couch_view:
>>    ./src/couchdb/couch_native_process.erl
>>    ./src/couchdb/couch_os_process.erl
>>    ./src/couchdb/couch_query_servers.erl
>>    ./src/couchdb/couch_view.erl
>>    ./src/couchdb/couch_view_compactor.erl
>>    ./src/couchdb/couch_view_group.erl
>>    ./src/couchdb/couch_view_updater.erl
>>    ./src/couchdb/couch_work_queue.erl
>>    ./src/couchdb/priv/couch_js/http.c
>>    ./src/couchdb/priv/couch_js/http.h
>>    ./src/couchdb/priv/couch_js/main.c
>>    ./src/couchdb/priv/couch_js/utf8.c
>>    ./src/couchdb/priv/couch_js/utf8.h
>>    ./src/couchdb/priv/spawnkillable/
>>    ./src/couchdb/priv/spawnkillable/couchspawnkillable_win.c
>> couch_rep:
>>    ./src/couchdb/couch_js_functions.hrl
>>    ./src/couchdb/couch_rep.erl
>>    ./src/couchdb/couch_rep_att.erl
>>    ./src/couchdb/couch_rep_changes_feed.erl
>>    ./src/couchdb/couch_rep_db_listener.erl
>>    ./src/couchdb/couch_rep_httpc.erl
>>    ./src/couchdb/couch_rep_missing_revs.erl
>>    ./src/couchdb/couch_rep_reader.erl
>>    ./src/couchdb/couch_rep_sup.erl
>>    ./src/couchdb/couch_rep_writer.erl
>> couch_externals:
>>    ./src/couchdb/couch_external_manager.erl
>>    ./src/couchdb/couch_external_server.erl
>> couch_httpd:
>>    ./src/couchdb/couch_httpd.erl
>>    ./src/couchdb/couch_httpd_auth.erl
>>    ./src/couchdb/couch_httpd_db.erl
>>    ./src/couchdb/couch_httpd_external.erl
>>    ./src/couchdb/couch_httpd_misc_handlers.erl
>>    ./src/couchdb/couch_httpd_oauth.erl
>>    ./src/couchdb/couch_httpd_rewrite.erl
>>    ./src/couchdb/couch_httpd_show.erl
>>    ./src/couchdb/couch_httpd_stats_handlers.erl
>>    ./src/couchdb/couch_httpd_vhost.erl
>>    ./src/couchdb/couch_httpd_view.erl

View raw message