couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Metson <si...@cloudant.com>
Subject Re: Minimally Viable Plugins
Date Sat, 03 Aug 2013 15:53:23 GMT
:)  


On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:

> Couldn’t help but implement it. It’s in the branch now.
>  
> Jan
> --  
>  
> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com> wrote:
>  
> > Sounds good to me.  
> >  
> >  
> > On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
> >  
> > >  
> > > On Aug 3, 2013, at 00:02 , Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)>
wrote:
> > >  
> > > > This is fantastic, Jan! Glad to see this coming along.
> > > >  
> > > > One of the goals with Fauxton has always been to make it easy for plugins
> > > > to extend the interface and provide new functionality. I've been toying
> > > > with the idea of having a _fauxton db that plugins install to as docs
with
> > > > attachments, but that's for a different thread. The general idea here
is
> > > > that a plugin will be able to extend Fauxton by adding a new page with
it's
> > > > own functionality, or hook into existing pages to extend other areas.
> > > >  
> > > > For instance, you could have a couchdb-lucene plugin that hooks into the
> > > > databases list and allows you to add interfaces for building full text
> > > > indexes and searching on existing indexes. Or you could have a dedicated
> > > > page for Geocouch, or whatever.
> > > >  
> > > > The functionality is there, but it's still a bit of a manual process,
so
> > > > we'll need to make it more dynamic and smooth out the rough edges.
> > > >  
> > > > I'm very excited to see progress being made on plugins, great work!
> > >  
> > > Thanks, I’m glad you like this! :)
> > >  
> > > Another way to get the Fauxton plugin loaded would be to extend the
> > > /_plugins API endpoint, so Fauxton could request GET /_plugins/<pluginname>/
> > > and it would serve <couchdblibdir>/plugins/<pluginname/priv/www which
is
> > > just a place for Fauxton-enabled plugins.
> > >  
> > > Fauxton would walk /_config/plugins/ to get to a list of plugins.
> > >  
> > > In fact that should be pretty simple to set up.
> > >  
> > > For now I am trying to avoid having a custom database for this, mostly
> > > because I don’t think there are many advantages (e.g. replication of
> > > plugins?) and code complexity. These priorities might change in the
> > > future, but for now I am happy to get this working at all :)
> > >  
> > > If you are okay with the above plan of serving plugin HTML/JS/CSS from
> > > /_plugins/<pluginname>, I’m happy to add this to the branch.
> > >  
> > > Best
> > > Jan
> > > --  
> > >  
> > >  
> > >  
> > > >  
> > > >  
> > > > -Russell
> > > >  
> > > >  
> > > > On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
> > > >  
> > > > > And a few more (from COUCHDB-1867):
> > > > >  
> > > > > - Add uninstall, incl. Futon UI.
> > > > > - Only install a plugin if the source and target CouchDB version
matches.
> > > > > - Rebase against master.
> > > > >  
> > > > > * * *
> > > > >  
> > > > > This concludes my list for a Minimally Viable Plugin feature. (See
the
> > > > > original email or README.md (http://README.md)* for the roadmap)
> > > > >  
> > > > > I’d appreciate some more reviews & feedback**, but other than
that, I’d be
> > > > > happy to ship this as an experimental feature in any next release.
> > > > >  
> > > > > *
> > > > > https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
> > > > > **
> > > > > https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
> > > > >  
> > > > >  
> > > > > Best
> > > > > Jan
> > > > > --
> > > > >  
> > > > > On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
> > > > >  
> > > > > > A few updates:
> > > > > >  
> > > > > > By Bob Ippolito / @etrepum:
> > > > > > - Plugins are now installed in libdir (instead of /tmp).
> > > > > > - Config loading is now done with proper .ini files.
> > > > > > - Various cleanups and code review (Thanks!).
> > > > > >  
> > > > > > Mine (most suggested by Bob):
> > > > > > - `plugins.html` now shows you if a plugin is already installed.
> > > > > > and which version, if it doesn’t match the installable one.
> > > > > > - The Install button now disables after an installation.
> > > > > > - Plugins are now registered with couch_config as
> > > > > > /_config/plugins/name = version
> > > > > > - Updated `couch-config` to print --erlang-version and --erl-bin
> > > > > > - Updated the geocouch plugin to use the new options in
> > > > > > `couch-config`.
> > > > > > - Added Bob Ippolito’s couchperuser plugin to Futon.
> > > > > >  
> > > > > >  
> > > > > > Best
> > > > > > Jan
> > > > > > --
> > > > > >  
> > > > > >  
> > > > > >  
> > > > > > On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan@apache.org
(mailto:jan@apache.org)> wrote:
> > > > > >  
> > > > > > > Heya,
> > > > > > >  
> > > > > > > I couldn’t help myself thinking about plugin stuff and
ended up
> > > > > > > whipping up a proof of concept.
> > > > > > >  
> > > > > > > Here’s a <1 minute demo video:
> > > > > > >  
> > > > > > > https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
> > > > > > >  
> > > > > > > Alternative encoding:
> > > > > > >  
> > > > > > > https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
> > > > > > >  
> > > > > > >  
> > > > > > > In my head the whole plugin idea is a very wide area, but
I was so
> > > > > > > intrigued by the idea of getting something running with
a click on a
> > > > > > > button in Futon. So I looked for a minimally viable plugin
system.
> > > > > > >  
> > > > > > >  
> > > > > > > ## Design principles
> > > > > > >  
> > > > > > > It took me a day to put this all together and this was
only possible
> > > > > > > because I took a lot of shortcuts. I believe they are all
viable for a
> > > > > > > first iteration of a plugins system:
> > > > > > >  
> > > > > > > 1. Install with one click on a button in Futon (or HTTP
call)
> > > > > > > 2. Only pure Erlang plugins are allowed.
> > > > > > > 3. The plugin author must provide a binary package for
each Erlang (and,
> > > > > > > later, each CouchDB version).
> > > > > > > 4. Complete trust-based system. You trust me to not do
any nasty things
> > > > > > > when you click on the install button. No crypto, no nothing.
Only
> > > > > > > people who can commit to Futon can release new versions
of plugins.
> > > > > > > 5. Minimal user-friendlyness: won’t install plugins that
don’t match
> > > > > > > the current Erlang version, gives semi-sensible error messages
> > > > > > > (wrapped in a HTTP 500 response :)
> > > > > > > 6. Require a pretty strict format for binary releases.
> > > > > > >  
> > > > > > >  
> > > > > > > ## Roadmap
> > > > > > >  
> > > > > > > Here’s a list of things this first iterations does and
doesn’t do:
> > > > > > >  
> > > > > > > - Pure Erlang plugins only. No C-dependencies, no JavaScript,
no
> > > > > nothing.
> > > > > > > - No C-dependencies.
> > > > > > > - Install a plugin via Futon (or HTTP call). Admin only.
> > > > > > > - A hardcoded list of plugins in Futon.
> > > > > > > - Loads a pre-packaged, pre-compiled .tar.gz file from
a URL.
> > > > > > > - Only installs if Erlang version matches.
> > > > > > > - No security checking of binaries.
> > > > > > > - No identity checking of binaries.
> > > > > > >  
> > > > > > > Here are a few things I want to add before I call it MVP*:
> > > > > > >  
> > > > > > > - Uninstall a plugin via Futon (or HTTP call). Admin only.
> > > > > > > - Only installs if CouchDB version matches.
> > > > > > > - Binaries must be published on *.apache.org (http://apache.org).
> > > > > > > - Register installed plugins in the config system.
> > > > > > > - Make sure plugins start with the next restart of CouchDB.
> > > > > > > - Show when a particular plugin is installed.
> > > > > > >  
> > > > > > > *MVP hopefully means you agree we can ship this with a
few warnings
> > > > > > > so people can get a hang of it.
> > > > > > >  
> > > > > > > Here is a rough list of features squared against future
milestones:
> > > > > > >  
> > > > > > > Milestone 2: Be creator friendly
> > > > > > > - Make it easy to build a CouchDB plugin by providing one
or more easy
> > > > > > > to start templates.
> > > > > > > - Make it easy to publish new plugins and new versions
of existing
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > plugins.
> > > > > > > - Make it easy to supply packages for multiple Erlang &
CouchDB
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > versions.
> > > > > > >  
> > > > > > > Milestone 3: Public registry
> > > > > > > - Instead of hardcoding a list of plugins into Futon/Fauxton,
we load
> > > > > > > a list of applicable plugins from a central (and configurable)
> > > > > > > plugins repository.
> > > > > > > - This allows plugin authors to publish new plugins and
new versions
> > > > > > > of existing plugins independently.
> > > > > > >  
> > > > > > > Milestone 4: Other Languages
> > > > > > > - Figure out how to handle C-dependencies for Erlang plugins.
> > > > > > > - Figure out how to allow other language plugins
> > > > > > > (c.f. non-JS query servers)
> > > > > > >  
> > > > > > > Milestone X: Later
> > > > > > > - Add some account/identity/maybe crypto-web-of-trust system
for
> > > > > > > authors to publish “legit” plugins.
> > > > > > > - Sign & verify individual releases.
> > > > > > >  
> > > > > > > A few more things that can happen concurrently depending
on what
> > > > > > > plugins require:
> > > > > > > - Integrate Erlang/JS tests in the installation
> > > > > > > - Integrate docs
> > > > > > >  
> > > > > > >  
> > > > > > > ## How it works
> > > > > > >  
> > > > > > > This plugin system lives in `src/couch_plugins` and is
a tiny CouchDB
> > > > > > > module.
> > > > > > >  
> > > > > > > It exposes one new API endpoint `/_plugins` that an admin
user can
> > > > > > > POST to.
> > > > > > >  
> > > > > > > The additional Futon page lives at /_utils/plugins.html
it is
> > > > > > > hardcoded.
> > > > > > >  
> > > > > > > Futon (or you) post an object to `/_plugins` with four
properties:
> > > > > > >  
> > > > > > > {
> > > > > > > "name": "geocouch", // name of the plugin, must be unique
> > > > > > > "url": "http://people.apache.org/~jan", // “base URL”
for plugin
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > releases (see below)
> > > > > > > "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever
version
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > internal to the plugin
> > > > > > > "checksums": {
> > > > > > > "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d
sha hash
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > over the binary
> > > > > > > }
> > > > > > > }
> > > > > > >  
> > > > > > > `couch_plugins` then attempts to download a .tar.gz from
this
> > > > > > > location:
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
> > > > > > >  
> > > > > > > It should be obvious how the URL is constructed from the
POST data.
> > > > > > > (This url is live, feel free to play around with this tarball).
> > > > > > >  
> > > > > > > Next it calculates the sha hash for the downloaded .tar.gz
file and
> > > > > > > matches it against the correct version in the `checksums`
parameter.
> > > > > > >  
> > > > > > > If that succeeds, we unpack the .tar.gz file (currently
in `/tmp`,
> > > > > > > need to find a better place for this) and adds the extracted
directory
> > > > > > > to the Erlang code path
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
> > > > > > > and loads the included application (`application:load(geocouch)`).
> > > > > > >  
> > > > > > > Then it looks into the `./config` directory that lives
next to `ebin/`
> > > > > > > in the plugin directory for a file `config.erlt` (“erl-terms”).
with a
> > > > > > > list of configuration parameters to load. We parse the
file and set
> > > > > > > the config directives one by one.
> > > > > > >  
> > > > > > > If that all goes to plan, we report success back to the
HTTP caller.
> > > > > > >  
> > > > > > > That’s it! :)
> > > > > > >  
> > > > > > > It’s deceptively simple, probably does a few things very
wrong and
> > > > > > > leaves a few things open (see above).
> > > > > > >  
> > > > > > > One open question I’d like an answer for is finding a
good location to
> > > > > > > unpack & install the plugin files that isn’t `tmp`.
If the answer is
> > > > > > > different for a pre-BigCouch/rcouch-merge and post-BigCouch/rcouch-
> > > > > > > merge world, I’d love to know :)
> > > > > > >  
> > > > > > >  
> > > > > > > ## Code
> > > > > > >  
> > > > > > > The main branch for this is 1867-feature-plugins:
> > > > > > >  
> > > > > > > ASF:
> > > > > https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
> > > > > > > GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
> > > > > > >  
> > > > > > > I created a branch on GeoCouch that adds a few lines to
its `Makefile`
> > > > > > > that shows how a binary package is built:
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > > >  
> > > > > https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
> > > > > > >  
> > > > > > > * * *
> > > > > > >  
> > > > > > > I hope you like this :) Please comment and improve heavily!
> > > > > > >  
> > > > > > > Let me know if you have any questions :)
> > > > > > >  
> > > > > > > If you have any criticism, please phrase it in a way that
we can use
> > > > > > > to improve this, thanks!
> > > > > > >  
> > > > > > > Best,
> > > > > > > Jan
> > > > > > > --
> > > > > > >  
> > > > > >  
> > > > >  
> > > > >  
> > > >  
> > > >  
> > >  
> > >  
> >  
> >  
>  
>  
>  



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message