incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Minimally Viable Plugins
Date Sun, 04 Aug 2013 15:39:38 GMT

On Aug 4, 2013, at 17:29 , Octavian Damiean <mainerror@gmail.com> wrote:

> Signed PGP part
> Great work!
> 
> Got a quick question. In the notice you mention that the instructions
> work only with the 1867-feature-plugin branch but you didn't add steps
> to checkout said branch in the preparation step.
> 
> Want me to add that bit of information or was that deliberate?

Total oversight, you are more then welcome to add this :)

Jan
--



> 
> Cheers,
> Octavian
> 
> On 2013-08-03 22:15, Jan Lehnardt wrote:
> > More update:
> > 
> > I started producing a plugin template repo that people can clone
> > to build their own plugins along with a comprehensive README.md:
> > 
> > https://github.com/janl/my-first-couchdb-plugin
> > 
> > The idea is to move the README to the CouchDB docs eventually and
> > ship the plugin template with CouchDB, so people can get started
> > easily.
> > 
> > Best Jan --
> > 
> > On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com>
> > wrote:
> > 
> >> :)
> >> 
> >> 
> >> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
> >> 
> >>> Couldn’t help but implement it. It’s in the branch now.
> >>> 
> >>> Jan --
> >>> 
> >>> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com>
> >>> wrote:
> >>> 
> >>>> Sounds good to me.
> >>>> 
> >>>> 
> >>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
> >>>> 
> >>>>> 
> >>>>> On Aug 3, 2013, at 00:02 , Russell Branca
> >>>>> <chewbranca@apache.org (mailto:chewbranca@apache.org)>
> >>>>> wrote:
> >>>>> 
> >>>>>> This is fantastic, Jan! Glad to see this coming along.
> >>>>>> 
> >>>>>> One of the goals with Fauxton has always been to make it
> >>>>>> easy for plugins to extend the interface and provide new
> >>>>>> functionality. I've been toying with the idea of having a
> >>>>>> _fauxton db that plugins install to as docs with 
> >>>>>> attachments, but that's for a different thread. The
> >>>>>> general idea here is that a plugin will be able to extend
> >>>>>> Fauxton by adding a new page with it's own functionality,
> >>>>>> or hook into existing pages to extend other areas.
> >>>>>> 
> >>>>>> For instance, you could have a couchdb-lucene plugin that
> >>>>>> hooks into the databases list and allows you to add
> >>>>>> interfaces for building full text indexes and searching
> >>>>>> on existing indexes. Or you could have a dedicated page
> >>>>>> for Geocouch, or whatever.
> >>>>>> 
> >>>>>> The functionality is there, but it's still a bit of a
> >>>>>> manual process, so we'll need to make it more dynamic and
> >>>>>> smooth out the rough edges.
> >>>>>> 
> >>>>>> I'm very excited to see progress being made on plugins,
> >>>>>> great work!
> >>>>> 
> >>>>> Thanks, I’m glad you like this! :)
> >>>>> 
> >>>>> Another way to get the Fauxton plugin loaded would be to
> >>>>> extend the /_plugins API endpoint, so Fauxton could request
> >>>>> GET /_plugins/<pluginname>/ and it would serve
> >>>>> <couchdblibdir>/plugins/<pluginname/priv/www which is just
> >>>>> a place for Fauxton-enabled plugins.
> >>>>> 
> >>>>> Fauxton would walk /_config/plugins/ to get to a list of
> >>>>> plugins.
> >>>>> 
> >>>>> In fact that should be pretty simple to set up.
> >>>>> 
> >>>>> For now I am trying to avoid having a custom database for
> >>>>> this, mostly because I don’t think there are many
> >>>>> advantages (e.g. replication of plugins?) and code
> >>>>> complexity. These priorities might change in the future,
> >>>>> but for now I am happy to get this working at all :)
> >>>>> 
> >>>>> If you are okay with the above plan of serving plugin
> >>>>> HTML/JS/CSS from /_plugins/<pluginname>, I’m happy to add
> >>>>> this to the branch.
> >>>>> 
> >>>>> Best Jan --
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> -Russell
> >>>>>> 
> >>>>>> 
> >>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt
> >>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
> >>>>>> 
> >>>>>>> And a few more (from COUCHDB-1867):
> >>>>>>> 
> >>>>>>> - Add uninstall, incl. Futon UI. - Only install a
> >>>>>>> plugin if the source and target CouchDB version
> >>>>>>> matches. - Rebase against master.
> >>>>>>> 
> >>>>>>> * * *
> >>>>>>> 
> >>>>>>> This concludes my list for a Minimally Viable Plugin
> >>>>>>> feature. (See the original email or README.md
> >>>>>>> (http://README.md)* for the roadmap)
> >>>>>>> 
> >>>>>>> I’d appreciate some more reviews & feedback**, but
> >>>>>>> other than that, I’d be happy to ship this as an
> >>>>>>> experimental feature in any next release.
> >>>>>>> 
> >>>>>>> * 
> >>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
> >>>>>>>
> >>>>>>> 
> **
> >>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> 
> Best
> >>>>>>> Jan --
> >>>>>>> 
> >>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org
> >>>>>>> (mailto:jan@apache.org)> wrote:
> >>>>>>> 
> >>>>>>>> A few updates:
> >>>>>>>> 
> >>>>>>>> By Bob Ippolito / @etrepum: - Plugins are now
> >>>>>>>> installed in libdir (instead of /tmp). - Config
> >>>>>>>> loading is now done with proper .ini files. - Various
> >>>>>>>> cleanups and code review (Thanks!).
> >>>>>>>> 
> >>>>>>>> Mine (most suggested by Bob): - `plugins.html` now
> >>>>>>>> shows you if a plugin is already installed. and which
> >>>>>>>> version, if it doesn’t match the installable one.
-
> >>>>>>>> The Install button now disables after an
> >>>>>>>> installation. - Plugins are now registered with
> >>>>>>>> couch_config as /_config/plugins/name = version -
> >>>>>>>> Updated `couch-config` to print --erlang-version and
> >>>>>>>> --erl-bin - Updated the geocouch plugin to use the
> >>>>>>>> new options in `couch-config`. - Added Bob Ippolito’s
> >>>>>>>> couchperuser plugin to Futon.
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> Best Jan --
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt
> >>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
> >>>>>>>> 
> >>>>>>>>> Heya,
> >>>>>>>>> 
> >>>>>>>>> I couldn’t help myself thinking about plugin stuff
> >>>>>>>>> and ended up whipping up a proof of concept.
> >>>>>>>>> 
> >>>>>>>>> Here’s a <1 minute demo video:
> >>>>>>>>> 
> >>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 
> Alternative encoding:
> >>>>>>>>> 
> >>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 
> In my head the whole plugin idea is a very wide area, but I was so
> >>>>>>>>> intrigued by the idea of getting something running
> >>>>>>>>> with a click on a button in Futon. So I looked for
> >>>>>>>>> a minimally viable plugin system.
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> ## Design principles
> >>>>>>>>> 
> >>>>>>>>> It took me a day to put this all together and this
> >>>>>>>>> was only possible because I took a lot of
> >>>>>>>>> shortcuts. I believe they are all viable for a 
> >>>>>>>>> first iteration of a plugins system:
> >>>>>>>>> 
> >>>>>>>>> 1. Install with one click on a button in Futon (or
> >>>>>>>>> HTTP call) 2. Only pure Erlang plugins are
> >>>>>>>>> allowed. 3. The plugin author must provide a binary
> >>>>>>>>> package for each Erlang (and, later, each CouchDB
> >>>>>>>>> version). 4. Complete trust-based system. You trust
> >>>>>>>>> me to not do any nasty things when you click on
the
> >>>>>>>>> install button. No crypto, no nothing. Only people
> >>>>>>>>> who can commit to Futon can release new versions
of
> >>>>>>>>> plugins. 5. Minimal user-friendlyness: won’t
> >>>>>>>>> install plugins that don’t match the current Erlang
> >>>>>>>>> version, gives semi-sensible error messages 
> >>>>>>>>> (wrapped in a HTTP 500 response :) 6. Require a
> >>>>>>>>> pretty strict format for binary releases.
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> ## Roadmap
> >>>>>>>>> 
> >>>>>>>>> Here’s a list of things this first iterations
does
> >>>>>>>>> and doesn’t do:
> >>>>>>>>> 
> >>>>>>>>> - Pure Erlang plugins only. No C-dependencies, no
> >>>>>>>>> JavaScript, no
> >>>>>>> nothing.
> >>>>>>>>> - No C-dependencies. - Install a plugin via Futon
> >>>>>>>>> (or HTTP call). Admin only. - A hardcoded list of
> >>>>>>>>> plugins in Futon. - Loads a pre-packaged,
> >>>>>>>>> pre-compiled .tar.gz file from a URL. - Only
> >>>>>>>>> installs if Erlang version matches. - No security
> >>>>>>>>> checking of binaries. - No identity checking of
> >>>>>>>>> binaries.
> >>>>>>>>> 
> >>>>>>>>> Here are a few things I want to add before I call
> >>>>>>>>> it MVP*:
> >>>>>>>>> 
> >>>>>>>>> - Uninstall a plugin via Futon (or HTTP call).
> >>>>>>>>> Admin only. - Only installs if CouchDB version
> >>>>>>>>> matches. - Binaries must be published on
> >>>>>>>>> *.apache.org (http://apache.org). - Register
> >>>>>>>>> installed plugins in the config system. - Make sure
> >>>>>>>>> plugins start with the next restart of CouchDB.
-
> >>>>>>>>> Show when a particular plugin is installed.
> >>>>>>>>> 
> >>>>>>>>> *MVP hopefully means you agree we can ship this
> >>>>>>>>> with a few warnings so people can get a hang of
> >>>>>>>>> it.
> >>>>>>>>> 
> >>>>>>>>> Here is a rough list of features squared against
> >>>>>>>>> future milestones:
> >>>>>>>>> 
> >>>>>>>>> Milestone 2: Be creator friendly - Make it easy
to
> >>>>>>>>> build a CouchDB plugin by providing one or more
> >>>>>>>>> easy to start templates. - Make it easy to publish
> >>>>>>>>> new plugins and new versions of existing
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> plugins.
> >>>>>>>>> - Make it easy to supply packages for multiple
> >>>>>>>>> Erlang & CouchDB
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> versions.
> >>>>>>>>> 
> >>>>>>>>> Milestone 3: Public registry - Instead of
> >>>>>>>>> hardcoding a list of plugins into Futon/Fauxton,
we
> >>>>>>>>> load a list of applicable plugins from a central
> >>>>>>>>> (and configurable) plugins repository. - This
> >>>>>>>>> allows plugin authors to publish new plugins and
> >>>>>>>>> new versions of existing plugins independently.
> >>>>>>>>> 
> >>>>>>>>> Milestone 4: Other Languages - Figure out how to
> >>>>>>>>> handle C-dependencies for Erlang plugins. - Figure
> >>>>>>>>> out how to allow other language plugins (c.f.
> >>>>>>>>> non-JS query servers)
> >>>>>>>>> 
> >>>>>>>>> Milestone X: Later - Add some
> >>>>>>>>> account/identity/maybe crypto-web-of-trust system
> >>>>>>>>> for authors to publish “legit” plugins. - Sign
&
> >>>>>>>>> verify individual releases.
> >>>>>>>>> 
> >>>>>>>>> A few more things that can happen concurrently
> >>>>>>>>> depending on what plugins require: - Integrate
> >>>>>>>>> Erlang/JS tests in the installation - Integrate
> >>>>>>>>> docs
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> ## How it works
> >>>>>>>>> 
> >>>>>>>>> This plugin system lives in `src/couch_plugins`
and
> >>>>>>>>> is a tiny CouchDB module.
> >>>>>>>>> 
> >>>>>>>>> It exposes one new API endpoint `/_plugins` that
an
> >>>>>>>>> admin user can POST to.
> >>>>>>>>> 
> >>>>>>>>> The additional Futon page lives at
> >>>>>>>>> /_utils/plugins.html it is hardcoded.
> >>>>>>>>> 
> >>>>>>>>> Futon (or you) post an object to `/_plugins` with
> >>>>>>>>> four properties:
> >>>>>>>>> 
> >>>>>>>>> { "name": "geocouch", // name of the plugin, must
> >>>>>>>>> be unique "url": "http://people.apache.org/~jan",
> >>>>>>>>> // “base URL” for plugin
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> releases (see below)
> >>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", //
> >>>>>>>>> whatever version
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> internal to the plugin
> >>>>>>>>> "checksums": { "R15B03":
> >>>>>>>>> "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d sha
> >>>>>>>>> hash
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> over the binary
> >>>>>>>>> } }
> >>>>>>>>> 
> >>>>>>>>> `couch_plugins` then attempts to download a .tar.gz
> >>>>>>>>> from this location:
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> 
> It should be obvious how the URL is constructed from the POST data.
> >>>>>>>>> (This url is live, feel free to play around with
> >>>>>>>>> this tarball).
> >>>>>>>>> 
> >>>>>>>>> Next it calculates the sha hash for the downloaded
> >>>>>>>>> .tar.gz file and matches it against the correct
> >>>>>>>>> version in the `checksums` parameter.
> >>>>>>>>> 
> >>>>>>>>> If that succeeds, we unpack the .tar.gz file
> >>>>>>>>> (currently in `/tmp`, need to find a better place
> >>>>>>>>> for this) and adds the extracted directory to the
> >>>>>>>>> Erlang code path
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
> >>>>>>>>>
> >>>>>>> 
> and loads the included application (`application:load(geocouch)`).
> >>>>>>>>> 
> >>>>>>>>> Then it looks into the `./config` directory that
> >>>>>>>>> lives next to `ebin/` in the plugin directory for
a
> >>>>>>>>> file `config.erlt` (“erl-terms”). with a list
of
> >>>>>>>>> configuration parameters to load. We parse the file
> >>>>>>>>> and set the config directives one by one.
> >>>>>>>>> 
> >>>>>>>>> If that all goes to plan, we report success back
to
> >>>>>>>>> the HTTP caller.
> >>>>>>>>> 
> >>>>>>>>> That’s it! :)
> >>>>>>>>> 
> >>>>>>>>> It’s deceptively simple, probably does a few things
> >>>>>>>>> very wrong and leaves a few things open (see
> >>>>>>>>> above).
> >>>>>>>>> 
> >>>>>>>>> One open question I’d like an answer for is finding
> >>>>>>>>> a good location to unpack & install the plugin
> >>>>>>>>> files that isn’t `tmp`. If the answer is different
> >>>>>>>>> for a pre-BigCouch/rcouch-merge and
> >>>>>>>>> post-BigCouch/rcouch- merge world, I’d love to
know
> >>>>>>>>> :)
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>> ## Code
> >>>>>>>>> 
> >>>>>>>>> The main branch for this is 1867-feature-plugins:
> >>>>>>>>> 
> >>>>>>>>> ASF:
> >>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
> >>>>>>>>>
> >>>>>>> 
> GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
> >>>>>>>>> 
> >>>>>>>>> I created a branch on GeoCouch that adds a few
> >>>>>>>>> lines to its `Makefile` that shows how a binary
> >>>>>>>>> package is built:
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>> 
> * * *
> >>>>>>>>> 
> >>>>>>>>> I hope you like this :) Please comment and improve
> >>>>>>>>> heavily!
> >>>>>>>>> 
> >>>>>>>>> Let me know if you have any questions :)
> >>>>>>>>> 
> >>>>>>>>> If you have any criticism, please phrase it in a
> >>>>>>>>> way that we can use to improve this, thanks!
> >>>>>>>>> 
> >>>>>>>>> Best, Jan --
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >>> 
> >> 
> >> 
> > 
> 


Mime
View raw message