couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob Dionne <>
Subject Re: CouchDB Module System
Date Mon, 23 Jan 2012 11:26:02 GMT

On Jan 23, 2012, at 4:33 AM, Paul Davis wrote:

> On Mon, Jan 23, 2012 at 3:03 AM, Randall Leeds <> wrote:
>> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <> wrote:
>>> Dear favorite Document-database developer community :-)
>>> This is my first message to the mailing list, so please forgive me and
>>> gently correct me if i am breaking any communication standards whatsoeverŠ
>> Nope! You're just bringing up a deep topic with a lot of text.
>> I hope you didn't expect a speedy response! :)
>>> I am currently working on a WebDAV interface for CouchDB, that now gets
>>> closer to its release date, somewhere in February or March and as i
>>> approach that, I want to package everything in a proper CouchDB Module
>>> format but the CouchDB Module system is, as i understood, in early
>>> development or pre-development phase and needs a bit of work.
>>> So as i work on a CouchDB Module anyways i would like to contribute to the
>>> Module system or start kicking the work on that of.
>>> Hence the following questions:
>>> Who started working on such a system or would be interested in talking
>>> about the design decisions that need to be done before actual coding can
>>> start?
>>> (I know at least Jan Lehnardt expressed interest, but i can't quite
>>> remember if he started actual work or just had plans for that.)
>> Benoit is particularly interested and there is activity on the refuge
>> project mailing list about it from Nicolas Dufour.
>> I believe Jason Smith might be pretty interested since he's authored
>> some "plugins" already.
>> Others on the list are certainly interested.
>> I'm keen to get a good module system and it's why I was in favor of
>> the effort to get "couch-config" shipped as part of the install.
>>> If there is any documentation on planning such a system, could someone
>>> please point me to the recourses?
>> There's an umbrella ticket I made for modularizing the code base more:
>> And these:
>> Utility to help plugin developers manage paths
>> support load of external erlang modules in couchdb.
>>> Lastly here is my two cents on a few topics that need to be addressed:
>>> - We should build the best module system there is for a database or
>>> content management system to gain back a bit of the traction we lost
>>> during the recent plateau phase of couchdb enthusiasm
>> I find it hard to judge enthusiasm from the inside, but, yeah... a
>> proper module system would rock.
>> One of the things CouchDB struggles with in terms of image, but excels
>> at in terms of function is scaling up and down. The only way to really
>> make one project move from mobile phone to datacenter cluster is to
>> modularize the build and deployment of key components. I think
>> everyone agrees about this. CouchDB is the Gentoo of document
>> databases in this way.
>>> - I think "module" is the preferable term, as extension, plugin or the
>>> like sound like second class citizens, whereas module implies, that even a
>>> core component of the system can be made into such an instance (Any other
>>> suggestions?)
>> Module is a little bit confusing because it conflicts with Erlang's
>> definition. While the two would often be synonymous for many add-ons,
>> it's not necessarily the case.
>> I would go with calling them "applications" maybe. I've often thought
>> that each such thing could correspond well to a single Erlang
>> application and .app file, which Erlang has significant support and
>> prior art for loading and unloading on the fly. But really, I don't
>> care what color we paint this and long as it's neon awesome.
>>> - A CouchDB module should be packaged as a stand alone couchdb-document,
>>> with the yet to be specified manifest beeing the json structure of the
>>> document and the files beeing the attachment.
>> I'm pretty sure I disagree here. I think this conflates two issues.
>> The first, to me, is a good hooks system from which more interactions
>> in the core could be scripted with sandboxed user code. Once again, I
>> know I've talked to Benoit about this. The second, though, is the more
>> first-class type of application that requires compiled Erlang
>> bytecode. While these have some degree of portability[1] I
>> instinctively cringe, from a security standpoint, at the idea of hot
>> loading code from a document into the VM. While the document-module
>> idea I find attractive from a share-ability and install-ability
>> standpoint, I don't think it befits the general purpose module system.
>> Although, a good hooks system may make certain kinds of modules work
>> this way.
>> [1]
>>> - Discovery, Installation, Enabling, Disabling, Module-Specific
>>> Configuration and Distribution should be part of Futon and introduce as
>>> little new interface concepts as possible, also the futon interface
>>> section "configuration" may be needed to be integrated and streamlined
>>> quite nicely
>> Again, this works the dynamic-language, hooks-based module, but I'm
>> not sure it works for all modules. Kanso [2] is doing this to a large
>> extent already, under the CouchApp frame, but I see no reason why a
>> hooks system couldn't accommodate a new, more powerful kind of Kanso
>> package.
>> [2]
>>> - Even if a module is not possible to completely install via this method
>>> (e.g. It depends on a os component, that is too low level etc.) such
>>> modules should be as tightly integrated into the system as possible and
>>> the manual admin installation steps should be documented in a way, so that
>>> in a normal case no place but the module system needs to be accessed
>> Agree.
>>> - on the top level the system should be the same for all OSes
>>> - The work packages i see at the moment for the structure i have in mind
>>> would be:
>>>        - Module repository: A CouchDB Instance, that hosts a collection of
>>> CouchDB Modules
>> See Kanso, above.
>>>        - Futon User Interface to browse available Modules in the configured
>>> repositories, should be close to the Firefox plugin paradigms
>> That would be neat. Or a separate 'shopping' utility for CouchApps.
>> Sounds a bit like a combination between JChris' CouchApp Garden [3]
>> and Max Ogden's Chrome Couch application [4]. Crossing these would
>> Kanso would be really neat.
>> [3]
>> [4]
>>>        - Module Manifest: Descriptive CouchDoc, that references the attached
>>> files or git repository, how files need to be fetched, packaged and what
>>> goes where, also version                dependencies, description, rating, Module
>>> Category, needed system privileges (basically if you need to be a couch
>>> admin or if you need to be root on the os), needed              configuration,
>>> basic description of the configuration parameters
>> I'd really prefer to punt on low-level privileges and just call this
>> two separate things. The way to unify them, in my mind, is to create a
>> hooks system and then to make a native module that listens for all
>> hooks and delegates processing to design documents that indicate they
>> handle certain hooks. In other words, build the un-privileged module
>> system on top of the privileged one. Let administrators worry about
>> file systems when installing the privileged ones. It's really not so
>> bad. Sysadmins have been installing things for years, and actually
>> probably prefer to install stuff on the command line than with a POST
>> to /_replicate. The couch-config utility is there to facilitate this.
>> What we probably need is an extra location for user-installed add-on
>> applications that need to be added to the Erlang library path and used
>> for loading configured applications.
>>>        - Futon User Interface to the installed modules, with options to enable,
>>> disable, uninstall, rate and configure
>> Yes again. I like the gist, but I don't know if it should be bound up
>> in Futon or not.
>>>        - The tools for installation/uninstallation and enabling/disabling as
>>> well as packaging and dependency resolution
>> That sounds like figuring out a good way to structure the
>> configuration sections for modules. See the JIRA tickets I referenced
>> for some discussion of that.
>>>        Open questions: What existing projects could be made use of, maybe there
>>> is something great for the general erlang world, maybe there is a project
>>> like zero install, that         could be made use of, maybe that is not
>>> necessary, what are your thoughts?
>> I would also check out Agner:
>> Basically, if we could get to the point where CouchDB administrators
>> could use Agner to install new pieces I'd consider that a huge win.
>> If, in the process, we get a modularized core and support for a
>> design-document-level hooks system I'd consider that an epic win. I
>> never say epic win. I would say it three times out loud.
>>> I wish you all a happy new year!
>> Thanks :)
>> -Randall
> +1 to pretty much everything Randall said.
> Minor caveat: application is about as overloaded in Erlang as module
> (and process for that matter) when considering prior expectations of
> non-Erlangians. There should be a word for these things. Plug-ins has
> always been my word for them, but the second-class-ness that invokes
> is understandable. Perhaps there's a perfect word, but perhaps
> "component" is good enough?
> My other reaction is that installing "whatevers/components/widgets"
> should *not* be a web thing. I understand the desire to make this user
> friendly and the such, but as Randall points out, putting random
> (fairly difficult to sandbox) code into a server/service environment
> seems quite ungood. This basically hearkens back to the
> disabled-by-default native view server. While its definitely
> "possible" to provide some sort of sandboxed thing for Erlang
> extensions, its a "plug all the leaks with fingers" type of possible.
> If we're interested in allowing code execution on the server it should
> be "here's you're iron box with carefully crafted ports of interaction
> to the world" type of sandboxing. Ie, AppEngine vs erm, not-AppEngine.
> Other than that, yes. Erlang makes extensions super awesomely easy and
> we should be embracing that more loudly. Both with community resources
> for discovery as well as the technical adjustments to ease their
> usage.

View raw message