Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: error (athena.apache.org: local policy)
From: Jan Lehnardt <jan@apache.org>
Content-Type: multipart/signed;
 boundary="Apple-Mail=_03DDBDEF-392C-4C83-B411-04CCF5D9CFC4";
 protocol="application/pgp-signature"; micalg=pgp-sha1
Message-Id: <6955D445-3D8A-482B-8F4F-12F94B1B12D0@apache.org>
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Minimally Viable Plugins
Date: Thu, 1 Aug 2013 19:34:43 +0200
References: <9BF26E7B-D6D4-4D1F-A11B-771038B7FACB@apache.org>
To: dev@couchdb.apache.org
In-Reply-To: <9BF26E7B-D6D4-4D1F-A11B-771038B7FACB@apache.org>

--Apple-Mail=_03DDBDEF-392C-4C83-B411-04CCF5D9CFC4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

A few updates:

By Bob Ippolito / @etrepum:
 - Plugins are now installed in libdir (instead of /tmp).
 - Config loading is now done with proper .ini files.
 - Various cleanups and code review (Thanks!).

Mine (most suggested by Bob):
 - `plugins.html` now shows you if a plugin is already installed.
   and which version, if it doesn=92t match the installable one.
 - The Install button now disables after an installation.
 - Plugins are now registered with couch_config as
   /_config/plugins/name =3D version
 - Updated `couch-config` to print --erlang-version and --erl-bin
 - Updated the geocouch plugin to use the new options in
   `couch-config`.
 - Added Bob Ippolito=92s couchperuser plugin to Futon.


Best
Jan
--=20


On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan@apache.org> wrote:

> Heya,
>=20
> I couldn=92t help myself thinking about plugin stuff and ended up
> whipping up a proof of concept.
>=20
> Here=92s a <1 minute demo video:
>=20
>  https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>=20
> Alternative encoding:
>=20
>  https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>=20
>=20
> In my head the whole plugin idea is a very wide area, but I was so
> intrigued by the idea of getting something running with a click on a
> button in Futon. So I looked for a minimally viable plugin system.
>=20
>=20
> ## Design principles
>=20
> It took me a day to put this all together and this was only possible
> because I took a lot of shortcuts. I believe they are all viable for a
> first iteration of a plugins system:
>=20
> 1. Install with one click on a button in Futon (or HTTP call)
> 2. Only pure Erlang plugins are allowed.
> 3. The plugin author must provide a binary package for each Erlang =
(and,
>   later, each CouchDB version).
> 4. Complete trust-based system. You trust me to not do any nasty =
things
>   when you click on the install button. No crypto, no nothing. Only
>   people who can commit to Futon can release new versions of plugins.
> 5. Minimal user-friendlyness: won=92t install plugins that don=92t =
match=20
>   the current Erlang version, gives semi-sensible error messages
>   (wrapped in a HTTP 500 response :)
> 6. Require a pretty strict format for binary releases.
>=20
>=20
> ## Roadmap
>=20
> Here=92s a list of things this first iterations does and doesn=92t do:
>=20
> - Pure Erlang plugins only. No C-dependencies, no JavaScript, no =
nothing.
> - No C-dependencies.
> - Install a plugin via Futon (or HTTP call). Admin only.
> - A hardcoded list of plugins in Futon.
> - Loads a pre-packaged, pre-compiled .tar.gz file from a URL.
> - Only installs if Erlang version matches.
> - No security checking of binaries.
> - No identity checking of binaries.
>=20
> Here are a few things I want to add before I call it MVP*:
>=20
> - Uninstall a plugin via Futon (or HTTP call). Admin only.
> - Only installs if CouchDB version matches.
> - Binaries must be published on *.apache.org.
> - Register installed plugins in the config system.
> - Make sure plugins start with the next restart of CouchDB.
> - Show when a particular plugin is installed.
>=20
> *MVP hopefully means you agree we can ship this with a few warnings
> so people can get a hang of it.
>=20
> Here is a rough list of features squared against future milestones:
>=20
> Milestone 2: Be creator friendly
> - Make it easy to build a CouchDB plugin by providing one or more easy=20=

>   to start templates.
> - Make it easy to publish new plugins and new versions of existing =
plugins.
> - Make it easy to supply packages for multiple Erlang & CouchDB =
versions.
>=20
> Milestone 3: Public registry
> - Instead of hardcoding a list of plugins into Futon/Fauxton, we load
>   a list of applicable plugins from a central (and configurable)
>   plugins repository.
> - This allows plugin authors to publish new plugins and new versions
>   of existing plugins independently.
>=20
> Milestone 4: Other Languages
> - Figure out how to handle C-dependencies for Erlang plugins.
> - Figure out how to allow other language plugins
>   (c.f. non-JS query servers)
>=20
> Milestone X: Later
> - Add some account/identity/maybe crypto-web-of-trust system for
>   authors to publish =93legit=94 plugins.
> - Sign & verify individual releases.
>=20
> A few more things that can happen concurrently depending on what
> plugins require:
> - Integrate Erlang/JS tests in the installation
> - Integrate docs
>=20
>=20
> ## How it works
>=20
> This plugin system lives in `src/couch_plugins` and is a tiny CouchDB
> module.
>=20
> It exposes one new API endpoint `/_plugins` that an admin user can
> POST to.
>=20
> The additional Futon page lives at /_utils/plugins.html it is
> hardcoded.
>=20
> Futon (or you) post an object to `/_plugins` with four properties:
>=20
>    {
>      "name": "geocouch", // name of the plugin, must be unique
>      "url": "http://people.apache.org/~jan", // =93base URL=94 for =
plugin releases (see below)
>      "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever version =
internal to the plugin
>      "checksums": {
>        "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=3D" // base64=92d sha =
hash over the binary
>      }
>    }
>=20
> `couch_plugins` then attempts to download a .tar.gz from this
> location:
>=20
>    =
http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15=
B03.tar.gz
>=20
> It should be obvious how the URL is constructed from the POST data.
> (This url is live, feel free to play around with this tarball).
>=20
> Next it calculates the sha hash for the downloaded .tar.gz file and
> matches it against the correct version in the `checksums` parameter.
>=20
> If that succeeds, we unpack the .tar.gz file (currently in `/tmp`,
> need to find a better place for this) and adds the extracted directory
> to the Erlang code path
> =
(`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea=
0bea-R15B03/ebin")`)
> and loads the included application (`application:load(geocouch)`).
>=20
> Then it looks into the `./config` directory that lives next to `ebin/`
> in the plugin directory for a file `config.erlt` (=93erl-terms=94). =
with a
> list of configuration parameters to load. We parse the file and set
> the config directives one by one.
>=20
> If that all goes to plan, we report success back to the HTTP caller.
>=20
> That=92s it! :)
>=20
> It=92s deceptively simple, probably does a few things very wrong and
> leaves a few things open (see above).
>=20
> One open question I=92d like an answer for is finding a good location =
to
> unpack & install the plugin files that isn=92t `tmp`. If the answer is
> different for a pre-BigCouch/rcouch-merge and post-BigCouch/rcouch-
> merge world, I=92d love to know :)
>=20
>=20
> ## Code
>=20
> The main branch for this is 1867-feature-plugins:
>=20
>     ASF: =
https://git-wip-us.apache.org/repos/asf?p=3Dcouchdb.git;a=3Dlog;h=3Drefs/h=
eads/1867-feature-plugins
>  GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>=20
> I created a branch on GeoCouch that adds a few lines to its `Makefile`
> that shows how a binary package is built:
>=20
>    =
https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1=
.3.x-plugins
>=20
> * * *
>=20
> I hope you like this :) Please comment and improve heavily!
>=20
> Let me know if you have any questions :)
>=20
> If you have any criticism, please phrase it in a way that we can use
> to improve this, thanks!
>=20
> Best,
> Jan
> --=20
>=20


--Apple-Mail=_03DDBDEF-392C-4C83-B411-04CCF5D9CFC4
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iEYEARECAAYFAlH6nDUACgkQ7KW8t7uWVrD3wQCfb28W/jlgofzE5GqFB1UJYVHS
OgcAoNRCeSbqUPy+5xcbuHwziL6QP9z0
=J9gI
-----END PGP SIGNATURE-----

--Apple-Mail=_03DDBDEF-392C-4C83-B411-04CCF5D9CFC4--