couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <>
Subject Re: CouchDB Module System
Date Mon, 23 Jan 2012 09:33:56 GMT
On Mon, Jan 23, 2012 at 3:03 AM, Randall Leeds <> wrote:
> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <> wrote:
>> Dear favorite Document-database developer community :-)
>> This is my first message to the mailing list, so please forgive me and
>> gently correct me if i am breaking any communication standards whatsoeverŠ
> Nope! You're just bringing up a deep topic with a lot of text.
> I hope you didn't expect a speedy response! :)
>> I am currently working on a WebDAV interface for CouchDB, that now gets
>> closer to its release date, somewhere in February or March and as i
>> approach that, I want to package everything in a proper CouchDB Module
>> format but the CouchDB Module system is, as i understood, in early
>> development or pre-development phase and needs a bit of work.
>> So as i work on a CouchDB Module anyways i would like to contribute to the
>> Module system or start kicking the work on that of.
>> Hence the following questions:
>> Who started working on such a system or would be interested in talking
>> about the design decisions that need to be done before actual coding can
>> start?
>> (I know at least Jan Lehnardt expressed interest, but i can't quite
>> remember if he started actual work or just had plans for that.)
> Benoit is particularly interested and there is activity on the refuge
> project mailing list about it from Nicolas Dufour.
> I believe Jason Smith might be pretty interested since he's authored
> some "plugins" already.
> Others on the list are certainly interested.
> I'm keen to get a good module system and it's why I was in favor of
> the effort to get "couch-config" shipped as part of the install.
>> If there is any documentation on planning such a system, could someone
>> please point me to the recourses?
> There's an umbrella ticket I made for modularizing the code base more:
> And these:
> Utility to help plugin developers manage paths
> support load of external erlang modules in couchdb.
>> Lastly here is my two cents on a few topics that need to be addressed:
>> - We should build the best module system there is for a database or
>> content management system to gain back a bit of the traction we lost
>> during the recent plateau phase of couchdb enthusiasm
> I find it hard to judge enthusiasm from the inside, but, yeah... a
> proper module system would rock.
> One of the things CouchDB struggles with in terms of image, but excels
> at in terms of function is scaling up and down. The only way to really
> make one project move from mobile phone to datacenter cluster is to
> modularize the build and deployment of key components. I think
> everyone agrees about this. CouchDB is the Gentoo of document
> databases in this way.
>> - I think "module" is the preferable term, as extension, plugin or the
>> like sound like second class citizens, whereas module implies, that even a
>> core component of the system can be made into such an instance (Any other
>> suggestions?)
> Module is a little bit confusing because it conflicts with Erlang's
> definition. While the two would often be synonymous for many add-ons,
> it's not necessarily the case.
> I would go with calling them "applications" maybe. I've often thought
> that each such thing could correspond well to a single Erlang
> application and .app file, which Erlang has significant support and
> prior art for loading and unloading on the fly. But really, I don't
> care what color we paint this and long as it's neon awesome.
>> - A CouchDB module should be packaged as a stand alone couchdb-document,
>> with the yet to be specified manifest beeing the json structure of the
>> document and the files beeing the attachment.
> I'm pretty sure I disagree here. I think this conflates two issues.
> The first, to me, is a good hooks system from which more interactions
> in the core could be scripted with sandboxed user code. Once again, I
> know I've talked to Benoit about this. The second, though, is the more
> first-class type of application that requires compiled Erlang
> bytecode. While these have some degree of portability[1] I
> instinctively cringe, from a security standpoint, at the idea of hot
> loading code from a document into the VM. While the document-module
> idea I find attractive from a share-ability and install-ability
> standpoint, I don't think it befits the general purpose module system.
> Although, a good hooks system may make certain kinds of modules work
> this way.
> [1]
>> - Discovery, Installation, Enabling, Disabling, Module-Specific
>> Configuration and Distribution should be part of Futon and introduce as
>> little new interface concepts as possible, also the futon interface
>> section "configuration" may be needed to be integrated and streamlined
>> quite nicely
> Again, this works the dynamic-language, hooks-based module, but I'm
> not sure it works for all modules. Kanso [2] is doing this to a large
> extent already, under the CouchApp frame, but I see no reason why a
> hooks system couldn't accommodate a new, more powerful kind of Kanso
> package.
> [2]
>> - Even if a module is not possible to completely install via this method
>> (e.g. It depends on a os component, that is too low level etc.) such
>> modules should be as tightly integrated into the system as possible and
>> the manual admin installation steps should be documented in a way, so that
>> in a normal case no place but the module system needs to be accessed
> Agree.
>> - on the top level the system should be the same for all OSes
>> - The work packages i see at the moment for the structure i have in mind
>> would be:
>>        - Module repository: A CouchDB Instance, that hosts a collection of
>> CouchDB Modules
> See Kanso, above.
>>        - Futon User Interface to browse available Modules in the configured
>> repositories, should be close to the Firefox plugin paradigms
> That would be neat. Or a separate 'shopping' utility for CouchApps.
> Sounds a bit like a combination between JChris' CouchApp Garden [3]
> and Max Ogden's Chrome Couch application [4]. Crossing these would
> Kanso would be really neat.
> [3]
> [4]
>>        - Module Manifest: Descriptive CouchDoc, that references the attached
>> files or git repository, how files need to be fetched, packaged and what
>> goes where, also version                dependencies, description, rating,
>> Category, needed system privileges (basically if you need to be a couch
>> admin or if you need to be root on the os), needed              configuration,
>> basic description of the configuration parameters
> I'd really prefer to punt on low-level privileges and just call this
> two separate things. The way to unify them, in my mind, is to create a
> hooks system and then to make a native module that listens for all
> hooks and delegates processing to design documents that indicate they
> handle certain hooks. In other words, build the un-privileged module
> system on top of the privileged one. Let administrators worry about
> file systems when installing the privileged ones. It's really not so
> bad. Sysadmins have been installing things for years, and actually
> probably prefer to install stuff on the command line than with a POST
> to /_replicate. The couch-config utility is there to facilitate this.
> What we probably need is an extra location for user-installed add-on
> applications that need to be added to the Erlang library path and used
> for loading configured applications.
>>        - Futon User Interface to the installed modules, with options to enable,
>> disable, uninstall, rate and configure
> Yes again. I like the gist, but I don't know if it should be bound up
> in Futon or not.
>>        - The tools for installation/uninstallation and enabling/disabling as
>> well as packaging and dependency resolution
> That sounds like figuring out a good way to structure the
> configuration sections for modules. See the JIRA tickets I referenced
> for some discussion of that.
>>        Open questions: What existing projects could be made use of, maybe there
>> is something great for the general erlang world, maybe there is a project
>> like zero install, that         could be made use of, maybe that is not
>> necessary, what are your thoughts?
> I would also check out Agner:
> Basically, if we could get to the point where CouchDB administrators
> could use Agner to install new pieces I'd consider that a huge win.
> If, in the process, we get a modularized core and support for a
> design-document-level hooks system I'd consider that an epic win. I
> never say epic win. I would say it three times out loud.
>> I wish you all a happy new year!
> Thanks :)
> -Randall

+1 to pretty much everything Randall said.

Minor caveat: application is about as overloaded in Erlang as module
(and process for that matter) when considering prior expectations of
non-Erlangians. There should be a word for these things. Plug-ins has
always been my word for them, but the second-class-ness that invokes
is understandable. Perhaps there's a perfect word, but perhaps
"component" is good enough?

My other reaction is that installing "whatevers/components/widgets"
should *not* be a web thing. I understand the desire to make this user
friendly and the such, but as Randall points out, putting random
(fairly difficult to sandbox) code into a server/service environment
seems quite ungood. This basically hearkens back to the
disabled-by-default native view server. While its definitely
"possible" to provide some sort of sandboxed thing for Erlang
extensions, its a "plug all the leaks with fingers" type of possible.
If we're interested in allowing code execution on the server it should
be "here's you're iron box with carefully crafted ports of interaction
to the world" type of sandboxing. Ie, AppEngine vs erm, not-AppEngine.

Other than that, yes. Erlang makes extensions super awesomely easy and
we should be embracing that more loudly. Both with community resources
for discovery as well as the technical adjustments to ease their

View raw message