couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Minimally Viable Plugins
Date Sat, 03 Aug 2013 13:21:21 GMT
Couldn’t help but implement it. It’s in the branch now.

Jan
-- 

On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com> wrote:

> Sounds good to me.  
> 
> 
> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
> 
>> 
>> On Aug 3, 2013, at 00:02 , Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)>
wrote:
>> 
>>> This is fantastic, Jan! Glad to see this coming along.
>>> 
>>> One of the goals with Fauxton has always been to make it easy for plugins
>>> to extend the interface and provide new functionality. I've been toying
>>> with the idea of having a _fauxton db that plugins install to as docs with
>>> attachments, but that's for a different thread. The general idea here is
>>> that a plugin will be able to extend Fauxton by adding a new page with it's
>>> own functionality, or hook into existing pages to extend other areas.
>>> 
>>> For instance, you could have a couchdb-lucene plugin that hooks into the
>>> databases list and allows you to add interfaces for building full text
>>> indexes and searching on existing indexes. Or you could have a dedicated
>>> page for Geocouch, or whatever.
>>> 
>>> The functionality is there, but it's still a bit of a manual process, so
>>> we'll need to make it more dynamic and smooth out the rough edges.
>>> 
>>> I'm very excited to see progress being made on plugins, great work!
>> 
>> Thanks, I’m glad you like this! :)
>> 
>> Another way to get the Fauxton plugin loaded would be to extend the
>> /_plugins API endpoint, so Fauxton could request GET /_plugins/<pluginname>/
>> and it would serve <couchdblibdir>/plugins/<pluginname/priv/www which is
>> just a place for Fauxton-enabled plugins.
>> 
>> Fauxton would walk /_config/plugins/ to get to a list of plugins.
>> 
>> In fact that should be pretty simple to set up.
>> 
>> For now I am trying to avoid having a custom database for this, mostly
>> because I don’t think there are many advantages (e.g. replication of
>> plugins?) and code complexity. These priorities might change in the
>> future, but for now I am happy to get this working at all :)
>> 
>> If you are okay with the above plan of serving plugin HTML/JS/CSS from
>> /_plugins/<pluginname>, I’m happy to add this to the branch.
>> 
>> Best
>> Jan
>> --  
>> 
>> 
>> 
>>> 
>>> 
>>> -Russell
>>> 
>>> 
>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>> 
>>>> And a few more (from COUCHDB-1867):
>>>> 
>>>> - Add uninstall, incl. Futon UI.
>>>> - Only install a plugin if the source and target CouchDB version matches.
>>>> - Rebase against master.
>>>> 
>>>> * * *
>>>> 
>>>> This concludes my list for a Minimally Viable Plugin feature. (See the
>>>> original email or README.md (http://README.md)* for the roadmap)
>>>> 
>>>> I’d appreciate some more reviews & feedback**, but other than that,
I’d be
>>>> happy to ship this as an experimental feature in any next release.
>>>> 
>>>> *
>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>>>> **
>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>>>> 
>>>> 
>>>> Best
>>>> Jan
>>>> --
>>>> 
>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>> 
>>>>> A few updates:
>>>>> 
>>>>> By Bob Ippolito / @etrepum:
>>>>> - Plugins are now installed in libdir (instead of /tmp).
>>>>> - Config loading is now done with proper .ini files.
>>>>> - Various cleanups and code review (Thanks!).
>>>>> 
>>>>> Mine (most suggested by Bob):
>>>>> - `plugins.html` now shows you if a plugin is already installed.
>>>>> and which version, if it doesn’t match the installable one.
>>>>> - The Install button now disables after an installation.
>>>>> - Plugins are now registered with couch_config as
>>>>> /_config/plugins/name = version
>>>>> - Updated `couch-config` to print --erlang-version and --erl-bin
>>>>> - Updated the geocouch plugin to use the new options in
>>>>> `couch-config`.
>>>>> - Added Bob Ippolito’s couchperuser plugin to Futon.
>>>>> 
>>>>> 
>>>>> Best
>>>>> Jan
>>>>> --
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>>> 
>>>>>> Heya,
>>>>>> 
>>>>>> I couldn’t help myself thinking about plugin stuff and ended up
>>>>>> whipping up a proof of concept.
>>>>>> 
>>>>>> Here’s a <1 minute demo video:
>>>>>> 
>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>>>>>> 
>>>>>> Alternative encoding:
>>>>>> 
>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>>>>>> 
>>>>>> 
>>>>>> In my head the whole plugin idea is a very wide area, but I was so
>>>>>> intrigued by the idea of getting something running with a click on
a
>>>>>> button in Futon. So I looked for a minimally viable plugin system.
>>>>>> 
>>>>>> 
>>>>>> ## Design principles
>>>>>> 
>>>>>> It took me a day to put this all together and this was only possible
>>>>>> because I took a lot of shortcuts. I believe they are all viable
for a
>>>>>> first iteration of a plugins system:
>>>>>> 
>>>>>> 1. Install with one click on a button in Futon (or HTTP call)
>>>>>> 2. Only pure Erlang plugins are allowed.
>>>>>> 3. The plugin author must provide a binary package for each Erlang
(and,
>>>>>> later, each CouchDB version).
>>>>>> 4. Complete trust-based system. You trust me to not do any nasty
things
>>>>>> when you click on the install button. No crypto, no nothing. Only
>>>>>> people who can commit to Futon can release new versions of plugins.
>>>>>> 5. Minimal user-friendlyness: won’t install plugins that don’t
match
>>>>>> the current Erlang version, gives semi-sensible error messages
>>>>>> (wrapped in a HTTP 500 response :)
>>>>>> 6. Require a pretty strict format for binary releases.
>>>>>> 
>>>>>> 
>>>>>> ## Roadmap
>>>>>> 
>>>>>> Here’s a list of things this first iterations does and doesn’t
do:
>>>>>> 
>>>>>> - Pure Erlang plugins only. No C-dependencies, no JavaScript, no
>>>> nothing.
>>>>>> - No C-dependencies.
>>>>>> - Install a plugin via Futon (or HTTP call). Admin only.
>>>>>> - A hardcoded list of plugins in Futon.
>>>>>> - Loads a pre-packaged, pre-compiled .tar.gz file from a URL.
>>>>>> - Only installs if Erlang version matches.
>>>>>> - No security checking of binaries.
>>>>>> - No identity checking of binaries.
>>>>>> 
>>>>>> Here are a few things I want to add before I call it MVP*:
>>>>>> 
>>>>>> - Uninstall a plugin via Futon (or HTTP call). Admin only.
>>>>>> - Only installs if CouchDB version matches.
>>>>>> - Binaries must be published on *.apache.org (http://apache.org).
>>>>>> - Register installed plugins in the config system.
>>>>>> - Make sure plugins start with the next restart of CouchDB.
>>>>>> - Show when a particular plugin is installed.
>>>>>> 
>>>>>> *MVP hopefully means you agree we can ship this with a few warnings
>>>>>> so people can get a hang of it.
>>>>>> 
>>>>>> Here is a rough list of features squared against future milestones:
>>>>>> 
>>>>>> Milestone 2: Be creator friendly
>>>>>> - Make it easy to build a CouchDB plugin by providing one or more
easy
>>>>>> to start templates.
>>>>>> - Make it easy to publish new plugins and new versions of existing
>>>>> 
>>>> 
>>>> 
>>>> plugins.
>>>>>> - Make it easy to supply packages for multiple Erlang & CouchDB
>>>>> 
>>>> 
>>>> 
>>>> versions.
>>>>>> 
>>>>>> Milestone 3: Public registry
>>>>>> - Instead of hardcoding a list of plugins into Futon/Fauxton, we
load
>>>>>> a list of applicable plugins from a central (and configurable)
>>>>>> plugins repository.
>>>>>> - This allows plugin authors to publish new plugins and new versions
>>>>>> of existing plugins independently.
>>>>>> 
>>>>>> Milestone 4: Other Languages
>>>>>> - Figure out how to handle C-dependencies for Erlang plugins.
>>>>>> - Figure out how to allow other language plugins
>>>>>> (c.f. non-JS query servers)
>>>>>> 
>>>>>> Milestone X: Later
>>>>>> - Add some account/identity/maybe crypto-web-of-trust system for
>>>>>> authors to publish “legit” plugins.
>>>>>> - Sign & verify individual releases.
>>>>>> 
>>>>>> A few more things that can happen concurrently depending on what
>>>>>> plugins require:
>>>>>> - Integrate Erlang/JS tests in the installation
>>>>>> - Integrate docs
>>>>>> 
>>>>>> 
>>>>>> ## How it works
>>>>>> 
>>>>>> This plugin system lives in `src/couch_plugins` and is a tiny CouchDB
>>>>>> module.
>>>>>> 
>>>>>> It exposes one new API endpoint `/_plugins` that an admin user can
>>>>>> POST to.
>>>>>> 
>>>>>> The additional Futon page lives at /_utils/plugins.html it is
>>>>>> hardcoded.
>>>>>> 
>>>>>> Futon (or you) post an object to `/_plugins` with four properties:
>>>>>> 
>>>>>> {
>>>>>> "name": "geocouch", // name of the plugin, must be unique
>>>>>> "url": "http://people.apache.org/~jan", // “base URL” for plugin
>>>>> 
>>>> 
>>>> 
>>>> releases (see below)
>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever version
>>>>> 
>>>> 
>>>> 
>>>> internal to the plugin
>>>>>> "checksums": {
>>>>>> "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d sha hash
>>>>> 
>>>> 
>>>> 
>>>> over the binary
>>>>>> }
>>>>>> }
>>>>>> 
>>>>>> `couch_plugins` then attempts to download a .tar.gz from this
>>>>>> location:
>>>>> 
>>>> 
>>>> 
>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>>>>>> 
>>>>>> It should be obvious how the URL is constructed from the POST data.
>>>>>> (This url is live, feel free to play around with this tarball).
>>>>>> 
>>>>>> Next it calculates the sha hash for the downloaded .tar.gz file and
>>>>>> matches it against the correct version in the `checksums` parameter.
>>>>>> 
>>>>>> If that succeeds, we unpack the .tar.gz file (currently in `/tmp`,
>>>>>> need to find a better place for this) and adds the extracted directory
>>>>>> to the Erlang code path
>>>>> 
>>>> 
>>>> 
>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>>>>>> and loads the included application (`application:load(geocouch)`).
>>>>>> 
>>>>>> Then it looks into the `./config` directory that lives next to `ebin/`
>>>>>> in the plugin directory for a file `config.erlt` (“erl-terms”).
with a
>>>>>> list of configuration parameters to load. We parse the file and set
>>>>>> the config directives one by one.
>>>>>> 
>>>>>> If that all goes to plan, we report success back to the HTTP caller.
>>>>>> 
>>>>>> That’s it! :)
>>>>>> 
>>>>>> It’s deceptively simple, probably does a few things very wrong
and
>>>>>> leaves a few things open (see above).
>>>>>> 
>>>>>> One open question I’d like an answer for is finding a good location
to
>>>>>> unpack & install the plugin files that isn’t `tmp`. If the
answer is
>>>>>> different for a pre-BigCouch/rcouch-merge and post-BigCouch/rcouch-
>>>>>> merge world, I’d love to know :)
>>>>>> 
>>>>>> 
>>>>>> ## Code
>>>>>> 
>>>>>> The main branch for this is 1867-feature-plugins:
>>>>>> 
>>>>>> ASF:
>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>>>>>> GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>>>>>> 
>>>>>> I created a branch on GeoCouch that adds a few lines to its `Makefile`
>>>>>> that shows how a binary package is built:
>>>>> 
>>>> 
>>>> 
>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>>>>>> 
>>>>>> * * *
>>>>>> 
>>>>>> I hope you like this :) Please comment and improve heavily!
>>>>>> 
>>>>>> Let me know if you have any questions :)
>>>>>> 
>>>>>> If you have any criticism, please phrase it in a way that we can
use
>>>>>> to improve this, thanks!
>>>>>> 
>>>>>> Best,
>>>>>> Jan
>>>>>> --
>>>>> 
>>>> 
>>> 
>> 
> 
> 
> 


Mime
View raw message