incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@apache.org>
Subject Re: Minimally Viable Plugins
Date Sat, 03 Aug 2013 21:03:48 GMT
Yis! <3


On 3 August 2013 21:15, Jan Lehnardt <jan@apache.org> wrote:

> More update:
>
> I started producing a plugin template repo that people can clone to
> build their own plugins along with a comprehensive README.md:
>
>     https://github.com/janl/my-first-couchdb-plugin
>
> The idea is to move the README to the CouchDB docs eventually and ship
> the plugin template with CouchDB, so people can get started easily.
>
> Best
> Jan
> --
>
> On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com> wrote:
>
> > :)
> >
> >
> > On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
> >
> >> Couldn’t help but implement it. It’s in the branch now.
> >>
> >> Jan
> >> --
> >>
> >> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com> wrote:
> >>
> >>> Sounds good to me.
> >>>
> >>>
> >>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
> >>>
> >>>>
> >>>> On Aug 3, 2013, at 00:02 , Russell Branca <chewbranca@apache.org(mailto:
> chewbranca@apache.org)> wrote:
> >>>>
> >>>>> This is fantastic, Jan! Glad to see this coming along.
> >>>>>
> >>>>> One of the goals with Fauxton has always been to make it easy for
> plugins
> >>>>> to extend the interface and provide new functionality. I've been
> toying
> >>>>> with the idea of having a _fauxton db that plugins install to as
> docs with
> >>>>> attachments, but that's for a different thread. The general idea
> here is
> >>>>> that a plugin will be able to extend Fauxton by adding a new page
> with it's
> >>>>> own functionality, or hook into existing pages to extend other areas.
> >>>>>
> >>>>> For instance, you could have a couchdb-lucene plugin that hooks
into
> the
> >>>>> databases list and allows you to add interfaces for building full
> text
> >>>>> indexes and searching on existing indexes. Or you could have a
> dedicated
> >>>>> page for Geocouch, or whatever.
> >>>>>
> >>>>> The functionality is there, but it's still a bit of a manual
> process, so
> >>>>> we'll need to make it more dynamic and smooth out the rough edges.
> >>>>>
> >>>>> I'm very excited to see progress being made on plugins, great work!
> >>>>
> >>>> Thanks, I’m glad you like this! :)
> >>>>
> >>>> Another way to get the Fauxton plugin loaded would be to extend the
> >>>> /_plugins API endpoint, so Fauxton could request GET
> /_plugins/<pluginname>/
> >>>> and it would serve <couchdblibdir>/plugins/<pluginname/priv/www
which
> is
> >>>> just a place for Fauxton-enabled plugins.
> >>>>
> >>>> Fauxton would walk /_config/plugins/ to get to a list of plugins.
> >>>>
> >>>> In fact that should be pretty simple to set up.
> >>>>
> >>>> For now I am trying to avoid having a custom database for this, mostly
> >>>> because I don’t think there are many advantages (e.g. replication
of
> >>>> plugins?) and code complexity. These priorities might change in the
> >>>> future, but for now I am happy to get this working at all :)
> >>>>
> >>>> If you are okay with the above plan of serving plugin HTML/JS/CSS from
> >>>> /_plugins/<pluginname>, I’m happy to add this to the branch.
> >>>>
> >>>> Best
> >>>> Jan
> >>>> --
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> -Russell
> >>>>>
> >>>>>
> >>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt <jan@apache.org(mailto:
> jan@apache.org)> wrote:
> >>>>>
> >>>>>> And a few more (from COUCHDB-1867):
> >>>>>>
> >>>>>> - Add uninstall, incl. Futon UI.
> >>>>>> - Only install a plugin if the source and target CouchDB version
> matches.
> >>>>>> - Rebase against master.
> >>>>>>
> >>>>>> * * *
> >>>>>>
> >>>>>> This concludes my list for a Minimally Viable Plugin feature.
(See
> the
> >>>>>> original email or README.md (http://README.md)* for the roadmap)
> >>>>>>
> >>>>>> I’d appreciate some more reviews & feedback**, but other
than that,
> I’d be
> >>>>>> happy to ship this as an experimental feature in any next release.
> >>>>>>
> >>>>>> *
> >>>>>>
> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
> >>>>>> **
> >>>>>>
> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
> >>>>>>
> >>>>>>
> >>>>>> Best
> >>>>>> Jan
> >>>>>> --
> >>>>>>
> >>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org (mailto:
> jan@apache.org)> wrote:
> >>>>>>
> >>>>>>> A few updates:
> >>>>>>>
> >>>>>>> By Bob Ippolito / @etrepum:
> >>>>>>> - Plugins are now installed in libdir (instead of /tmp).
> >>>>>>> - Config loading is now done with proper .ini files.
> >>>>>>> - Various cleanups and code review (Thanks!).
> >>>>>>>
> >>>>>>> Mine (most suggested by Bob):
> >>>>>>> - `plugins.html` now shows you if a plugin is already installed.
> >>>>>>> and which version, if it doesn’t match the installable
one.
> >>>>>>> - The Install button now disables after an installation.
> >>>>>>> - Plugins are now registered with couch_config as
> >>>>>>> /_config/plugins/name = version
> >>>>>>> - Updated `couch-config` to print --erlang-version and --erl-bin
> >>>>>>> - Updated the geocouch plugin to use the new options in
> >>>>>>> `couch-config`.
> >>>>>>> - Added Bob Ippolito’s couchperuser plugin to Futon.
> >>>>>>>
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Jan
> >>>>>>> --
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan@apache.org
(mailto:
> jan@apache.org)> wrote:
> >>>>>>>
> >>>>>>>> Heya,
> >>>>>>>>
> >>>>>>>> I couldn’t help myself thinking about plugin stuff
and ended up
> >>>>>>>> whipping up a proof of concept.
> >>>>>>>>
> >>>>>>>> Here’s a <1 minute demo video:
> >>>>>>>>
> >>>>>>>>
> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
> >>>>>>>>
> >>>>>>>> Alternative encoding:
> >>>>>>>>
> >>>>>>>>
> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In my head the whole plugin idea is a very wide area,
but I was so
> >>>>>>>> intrigued by the idea of getting something running with
a click
> on a
> >>>>>>>> button in Futon. So I looked for a minimally viable
plugin system.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ## Design principles
> >>>>>>>>
> >>>>>>>> It took me a day to put this all together and this was
only
> possible
> >>>>>>>> because I took a lot of shortcuts. I believe they are
all viable
> for a
> >>>>>>>> first iteration of a plugins system:
> >>>>>>>>
> >>>>>>>> 1. Install with one click on a button in Futon (or HTTP
call)
> >>>>>>>> 2. Only pure Erlang plugins are allowed.
> >>>>>>>> 3. The plugin author must provide a binary package for
each
> Erlang (and,
> >>>>>>>> later, each CouchDB version).
> >>>>>>>> 4. Complete trust-based system. You trust me to not
do any nasty
> things
> >>>>>>>> when you click on the install button. No crypto, no
nothing. Only
> >>>>>>>> people who can commit to Futon can release new versions
of
> plugins.
> >>>>>>>> 5. Minimal user-friendlyness: won’t install plugins
that don’t
> match
> >>>>>>>> the current Erlang version, gives semi-sensible error
messages
> >>>>>>>> (wrapped in a HTTP 500 response :)
> >>>>>>>> 6. Require a pretty strict format for binary releases.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ## Roadmap
> >>>>>>>>
> >>>>>>>> Here’s a list of things this first iterations does
and doesn’t do:
> >>>>>>>>
> >>>>>>>> - Pure Erlang plugins only. No C-dependencies, no JavaScript,
no
> >>>>>> nothing.
> >>>>>>>> - No C-dependencies.
> >>>>>>>> - Install a plugin via Futon (or HTTP call). Admin only.
> >>>>>>>> - A hardcoded list of plugins in Futon.
> >>>>>>>> - Loads a pre-packaged, pre-compiled .tar.gz file from
a URL.
> >>>>>>>> - Only installs if Erlang version matches.
> >>>>>>>> - No security checking of binaries.
> >>>>>>>> - No identity checking of binaries.
> >>>>>>>>
> >>>>>>>> Here are a few things I want to add before I call it
MVP*:
> >>>>>>>>
> >>>>>>>> - Uninstall a plugin via Futon (or HTTP call). Admin
only.
> >>>>>>>> - Only installs if CouchDB version matches.
> >>>>>>>> - Binaries must be published on *.apache.org (http://apache.org).
> >>>>>>>> - Register installed plugins in the config system.
> >>>>>>>> - Make sure plugins start with the next restart of CouchDB.
> >>>>>>>> - Show when a particular plugin is installed.
> >>>>>>>>
> >>>>>>>> *MVP hopefully means you agree we can ship this with
a few
> warnings
> >>>>>>>> so people can get a hang of it.
> >>>>>>>>
> >>>>>>>> Here is a rough list of features squared against future
> milestones:
> >>>>>>>>
> >>>>>>>> Milestone 2: Be creator friendly
> >>>>>>>> - Make it easy to build a CouchDB plugin by providing
one or more
> easy
> >>>>>>>> to start templates.
> >>>>>>>> - Make it easy to publish new plugins and new versions
of existing
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> plugins.
> >>>>>>>> - Make it easy to supply packages for multiple Erlang
& CouchDB
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> versions.
> >>>>>>>>
> >>>>>>>> Milestone 3: Public registry
> >>>>>>>> - Instead of hardcoding a list of plugins into Futon/Fauxton,
we
> load
> >>>>>>>> a list of applicable plugins from a central (and configurable)
> >>>>>>>> plugins repository.
> >>>>>>>> - This allows plugin authors to publish new plugins
and new
> versions
> >>>>>>>> of existing plugins independently.
> >>>>>>>>
> >>>>>>>> Milestone 4: Other Languages
> >>>>>>>> - Figure out how to handle C-dependencies for Erlang
plugins.
> >>>>>>>> - Figure out how to allow other language plugins
> >>>>>>>> (c.f. non-JS query servers)
> >>>>>>>>
> >>>>>>>> Milestone X: Later
> >>>>>>>> - Add some account/identity/maybe crypto-web-of-trust
system for
> >>>>>>>> authors to publish “legit” plugins.
> >>>>>>>> - Sign & verify individual releases.
> >>>>>>>>
> >>>>>>>> A few more things that can happen concurrently depending
on what
> >>>>>>>> plugins require:
> >>>>>>>> - Integrate Erlang/JS tests in the installation
> >>>>>>>> - Integrate docs
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ## How it works
> >>>>>>>>
> >>>>>>>> This plugin system lives in `src/couch_plugins` and
is a tiny
> CouchDB
> >>>>>>>> module.
> >>>>>>>>
> >>>>>>>> It exposes one new API endpoint `/_plugins` that an
admin user can
> >>>>>>>> POST to.
> >>>>>>>>
> >>>>>>>> The additional Futon page lives at /_utils/plugins.html
it is
> >>>>>>>> hardcoded.
> >>>>>>>>
> >>>>>>>> Futon (or you) post an object to `/_plugins` with four
properties:
> >>>>>>>>
> >>>>>>>> {
> >>>>>>>> "name": "geocouch", // name of the plugin, must be unique
> >>>>>>>> "url": "http://people.apache.org/~jan", // “base URL”
for plugin
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> releases (see below)
> >>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever
version
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> internal to the plugin
> >>>>>>>> "checksums": {
> >>>>>>>> "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d
sha hash
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> over the binary
> >>>>>>>> }
> >>>>>>>> }
> >>>>>>>>
> >>>>>>>> `couch_plugins` then attempts to download a .tar.gz
from this
> >>>>>>>> location:
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
> >>>>>>>>
> >>>>>>>> It should be obvious how the URL is constructed from
the POST
> data.
> >>>>>>>> (This url is live, feel free to play around with this
tarball).
> >>>>>>>>
> >>>>>>>> Next it calculates the sha hash for the downloaded .tar.gz
file
> and
> >>>>>>>> matches it against the correct version in the `checksums`
> parameter.
> >>>>>>>>
> >>>>>>>> If that succeeds, we unpack the .tar.gz file (currently
in `/tmp`,
> >>>>>>>> need to find a better place for this) and adds the extracted
> directory
> >>>>>>>> to the Erlang code path
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
> >>>>>>>> and loads the included application (`application:load(geocouch)`).
> >>>>>>>>
> >>>>>>>> Then it looks into the `./config` directory that lives
next to
> `ebin/`
> >>>>>>>> in the plugin directory for a file `config.erlt` (“erl-terms”).
> with a
> >>>>>>>> list of configuration parameters to load. We parse the
file and
> set
> >>>>>>>> the config directives one by one.
> >>>>>>>>
> >>>>>>>> If that all goes to plan, we report success back to
the HTTP
> caller.
> >>>>>>>>
> >>>>>>>> That’s it! :)
> >>>>>>>>
> >>>>>>>> It’s deceptively simple, probably does a few things
very wrong and
> >>>>>>>> leaves a few things open (see above).
> >>>>>>>>
> >>>>>>>> One open question I’d like an answer for is finding
a good
> location to
> >>>>>>>> unpack & install the plugin files that isn’t `tmp`.
If the answer
> is
> >>>>>>>> different for a pre-BigCouch/rcouch-merge and
> post-BigCouch/rcouch-
> >>>>>>>> merge world, I’d love to know :)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ## Code
> >>>>>>>>
> >>>>>>>> The main branch for this is 1867-feature-plugins:
> >>>>>>>>
> >>>>>>>> ASF:
> >>>>>>
> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
> >>>>>>>> GitHub:
> https://github.com/janl/couchdb/compare/1867-feature-plugins
> >>>>>>>>
> >>>>>>>> I created a branch on GeoCouch that adds a few lines
to its
> `Makefile`
> >>>>>>>> that shows how a binary package is built:
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
> >>>>>>>>
> >>>>>>>> * * *
> >>>>>>>>
> >>>>>>>> I hope you like this :) Please comment and improve heavily!
> >>>>>>>>
> >>>>>>>> Let me know if you have any questions :)
> >>>>>>>>
> >>>>>>>> If you have any criticism, please phrase it in a way
that we can
> use
> >>>>>>>> to improve this, thanks!
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Jan
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
>
>


-- 
Noah Slater
https://twitter.com/nslater

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message