Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9B0A41023B for ; Sat, 3 Aug 2013 20:16:25 +0000 (UTC) Received: (qmail 91220 invoked by uid 500); 3 Aug 2013 20:16:24 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 91190 invoked by uid 500); 3 Aug 2013 20:16:24 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 91182 invoked by uid 99); 3 Aug 2013 20:16:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Aug 2013 20:16:24 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [80.244.253.218] (HELO mail.traeumt.net) (80.244.253.218) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Aug 2013 20:16:20 +0000 Received: from [192.168.0.195] (24-134-29-37-dynip.superkabel.de [24.134.29.37]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.traeumt.net (Postfix) with ESMTPSA id 660CA142A7 for ; Sat, 3 Aug 2013 22:17:43 +0200 (CEST) From: Jan Lehnardt Content-Type: multipart/signed; boundary="Apple-Mail=_00CCB7F7-801F-4782-BC00-0577EAA7010D"; protocol="application/pgp-signature"; micalg=pgp-sha1 Message-Id: <20E0B52F-5B43-4E16-AAB6-3DA495F6261B@apache.org> Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: Minimally Viable Plugins Date: Sat, 3 Aug 2013 22:15:37 +0200 References: <9BF26E7B-D6D4-4D1F-A11B-771038B7FACB@apache.org> <6955D445-3D8A-482B-8F4F-12F94B1B12D0@apache.org> <3A847F88-3E70-4463-9E91-383290D871F5@apache.org> <3754CC5CF2D84A32B807CB0E681AFDE4@cloudant.com> To: dev@couchdb.apache.org In-Reply-To: <3754CC5CF2D84A32B807CB0E681AFDE4@cloudant.com> X-Mailer: Apple Mail (2.1508) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_00CCB7F7-801F-4782-BC00-0577EAA7010D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 More update: I started producing a plugin template repo that people can clone to build their own plugins along with a comprehensive README.md: https://github.com/janl/my-first-couchdb-plugin The idea is to move the README to the CouchDB docs eventually and ship the plugin template with CouchDB, so people can get started easily. Best Jan -- On Aug 3, 2013, at 17:53 , Simon Metson wrote: > :) =20 >=20 >=20 > On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote: >=20 >> Couldn=92t help but implement it. It=92s in the branch now. >>=20 >> Jan >> -- =20 >>=20 >> On Aug 3, 2013, at 08:12 , Simon Metson wrote: >>=20 >>> Sounds good to me. =20 >>>=20 >>>=20 >>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote: >>>=20 >>>>=20 >>>> On Aug 3, 2013, at 00:02 , Russell Branca wrote: >>>>=20 >>>>> This is fantastic, Jan! Glad to see this coming along. >>>>>=20 >>>>> One of the goals with Fauxton has always been to make it easy for = plugins >>>>> to extend the interface and provide new functionality. I've been = toying >>>>> with the idea of having a _fauxton db that plugins install to as = docs with >>>>> attachments, but that's for a different thread. The general idea = here is >>>>> that a plugin will be able to extend Fauxton by adding a new page = with it's >>>>> own functionality, or hook into existing pages to extend other = areas. >>>>>=20 >>>>> For instance, you could have a couchdb-lucene plugin that hooks = into the >>>>> databases list and allows you to add interfaces for building full = text >>>>> indexes and searching on existing indexes. Or you could have a = dedicated >>>>> page for Geocouch, or whatever. >>>>>=20 >>>>> The functionality is there, but it's still a bit of a manual = process, so >>>>> we'll need to make it more dynamic and smooth out the rough edges. >>>>>=20 >>>>> I'm very excited to see progress being made on plugins, great = work! >>>>=20 >>>> Thanks, I=92m glad you like this! :) >>>>=20 >>>> Another way to get the Fauxton plugin loaded would be to extend the >>>> /_plugins API endpoint, so Fauxton could request GET = /_plugins// >>>> and it would serve /plugins/>>> just a place for Fauxton-enabled plugins. >>>>=20 >>>> Fauxton would walk /_config/plugins/ to get to a list of plugins. >>>>=20 >>>> In fact that should be pretty simple to set up. >>>>=20 >>>> For now I am trying to avoid having a custom database for this, = mostly >>>> because I don=92t think there are many advantages (e.g. replication = of >>>> plugins?) and code complexity. These priorities might change in the >>>> future, but for now I am happy to get this working at all :) >>>>=20 >>>> If you are okay with the above plan of serving plugin HTML/JS/CSS = from >>>> /_plugins/, I=92m happy to add this to the branch. >>>>=20 >>>> Best >>>> Jan >>>> -- =20 >>>>=20 >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>> -Russell >>>>>=20 >>>>>=20 >>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt wrote: >>>>>=20 >>>>>> And a few more (from COUCHDB-1867): >>>>>>=20 >>>>>> - Add uninstall, incl. Futon UI. >>>>>> - Only install a plugin if the source and target CouchDB version = matches. >>>>>> - Rebase against master. >>>>>>=20 >>>>>> * * * >>>>>>=20 >>>>>> This concludes my list for a Minimally Viable Plugin feature. = (See the >>>>>> original email or README.md (http://README.md)* for the roadmap) >>>>>>=20 >>>>>> I=92d appreciate some more reviews & feedback**, but other than = that, I=92d be >>>>>> happy to ship this as an experimental feature in any next = release. >>>>>>=20 >>>>>> * >>>>>> = https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugin= s/README.md#roadmap >>>>>> ** >>>>>> = https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugi= ns >>>>>>=20 >>>>>>=20 >>>>>> Best >>>>>> Jan >>>>>> -- >>>>>>=20 >>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt wrote: >>>>>>=20 >>>>>>> A few updates: >>>>>>>=20 >>>>>>> By Bob Ippolito / @etrepum: >>>>>>> - Plugins are now installed in libdir (instead of /tmp). >>>>>>> - Config loading is now done with proper .ini files. >>>>>>> - Various cleanups and code review (Thanks!). >>>>>>>=20 >>>>>>> Mine (most suggested by Bob): >>>>>>> - `plugins.html` now shows you if a plugin is already installed. >>>>>>> and which version, if it doesn=92t match the installable one. >>>>>>> - The Install button now disables after an installation. >>>>>>> - Plugins are now registered with couch_config as >>>>>>> /_config/plugins/name =3D version >>>>>>> - Updated `couch-config` to print --erlang-version and --erl-bin >>>>>>> - Updated the geocouch plugin to use the new options in >>>>>>> `couch-config`. >>>>>>> - Added Bob Ippolito=92s couchperuser plugin to Futon. >>>>>>>=20 >>>>>>>=20 >>>>>>> Best >>>>>>> Jan >>>>>>> -- >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt wrote: >>>>>>>=20 >>>>>>>> Heya, >>>>>>>>=20 >>>>>>>> I couldn=92t help myself thinking about plugin stuff and ended = up >>>>>>>> whipping up a proof of concept. >>>>>>>>=20 >>>>>>>> Here=92s a <1 minute demo video: >>>>>>>>=20 >>>>>>>> = https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov >>>>>>>>=20 >>>>>>>> Alternative encoding: >>>>>>>>=20 >>>>>>>> = https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v) >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> In my head the whole plugin idea is a very wide area, but I was = so >>>>>>>> intrigued by the idea of getting something running with a click = on a >>>>>>>> button in Futon. So I looked for a minimally viable plugin = system. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ## Design principles >>>>>>>>=20 >>>>>>>> It took me a day to put this all together and this was only = possible >>>>>>>> because I took a lot of shortcuts. I believe they are all = viable for a >>>>>>>> first iteration of a plugins system: >>>>>>>>=20 >>>>>>>> 1. Install with one click on a button in Futon (or HTTP call) >>>>>>>> 2. Only pure Erlang plugins are allowed. >>>>>>>> 3. The plugin author must provide a binary package for each = Erlang (and, >>>>>>>> later, each CouchDB version). >>>>>>>> 4. Complete trust-based system. You trust me to not do any = nasty things >>>>>>>> when you click on the install button. No crypto, no nothing. = Only >>>>>>>> people who can commit to Futon can release new versions of = plugins. >>>>>>>> 5. Minimal user-friendlyness: won=92t install plugins that = don=92t match >>>>>>>> the current Erlang version, gives semi-sensible error messages >>>>>>>> (wrapped in a HTTP 500 response :) >>>>>>>> 6. Require a pretty strict format for binary releases. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ## Roadmap >>>>>>>>=20 >>>>>>>> Here=92s a list of things this first iterations does and = doesn=92t do: >>>>>>>>=20 >>>>>>>> - Pure Erlang plugins only. No C-dependencies, no JavaScript, = no >>>>>> nothing. >>>>>>>> - No C-dependencies. >>>>>>>> - Install a plugin via Futon (or HTTP call). Admin only. >>>>>>>> - A hardcoded list of plugins in Futon. >>>>>>>> - Loads a pre-packaged, pre-compiled .tar.gz file from a URL. >>>>>>>> - Only installs if Erlang version matches. >>>>>>>> - No security checking of binaries. >>>>>>>> - No identity checking of binaries. >>>>>>>>=20 >>>>>>>> Here are a few things I want to add before I call it MVP*: >>>>>>>>=20 >>>>>>>> - Uninstall a plugin via Futon (or HTTP call). Admin only. >>>>>>>> - Only installs if CouchDB version matches. >>>>>>>> - Binaries must be published on *.apache.org = (http://apache.org). >>>>>>>> - Register installed plugins in the config system. >>>>>>>> - Make sure plugins start with the next restart of CouchDB. >>>>>>>> - Show when a particular plugin is installed. >>>>>>>>=20 >>>>>>>> *MVP hopefully means you agree we can ship this with a few = warnings >>>>>>>> so people can get a hang of it. >>>>>>>>=20 >>>>>>>> Here is a rough list of features squared against future = milestones: >>>>>>>>=20 >>>>>>>> Milestone 2: Be creator friendly >>>>>>>> - Make it easy to build a CouchDB plugin by providing one or = more easy >>>>>>>> to start templates. >>>>>>>> - Make it easy to publish new plugins and new versions of = existing >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> plugins. >>>>>>>> - Make it easy to supply packages for multiple Erlang & CouchDB >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> versions. >>>>>>>>=20 >>>>>>>> Milestone 3: Public registry >>>>>>>> - Instead of hardcoding a list of plugins into Futon/Fauxton, = we load >>>>>>>> a list of applicable plugins from a central (and configurable) >>>>>>>> plugins repository. >>>>>>>> - This allows plugin authors to publish new plugins and new = versions >>>>>>>> of existing plugins independently. >>>>>>>>=20 >>>>>>>> Milestone 4: Other Languages >>>>>>>> - Figure out how to handle C-dependencies for Erlang plugins. >>>>>>>> - Figure out how to allow other language plugins >>>>>>>> (c.f. non-JS query servers) >>>>>>>>=20 >>>>>>>> Milestone X: Later >>>>>>>> - Add some account/identity/maybe crypto-web-of-trust system = for >>>>>>>> authors to publish =93legit=94 plugins. >>>>>>>> - Sign & verify individual releases. >>>>>>>>=20 >>>>>>>> A few more things that can happen concurrently depending on = what >>>>>>>> plugins require: >>>>>>>> - Integrate Erlang/JS tests in the installation >>>>>>>> - Integrate docs >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ## How it works >>>>>>>>=20 >>>>>>>> This plugin system lives in `src/couch_plugins` and is a tiny = CouchDB >>>>>>>> module. >>>>>>>>=20 >>>>>>>> It exposes one new API endpoint `/_plugins` that an admin user = can >>>>>>>> POST to. >>>>>>>>=20 >>>>>>>> The additional Futon page lives at /_utils/plugins.html it is >>>>>>>> hardcoded. >>>>>>>>=20 >>>>>>>> Futon (or you) post an object to `/_plugins` with four = properties: >>>>>>>>=20 >>>>>>>> { >>>>>>>> "name": "geocouch", // name of the plugin, must be unique >>>>>>>> "url": "http://people.apache.org/~jan", // =93base URL=94 for = plugin >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> releases (see below) >>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever = version >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> internal to the plugin >>>>>>>> "checksums": { >>>>>>>> "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=3D" // base64=92d sha = hash >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> over the binary >>>>>>>> } >>>>>>>> } >>>>>>>>=20 >>>>>>>> `couch_plugins` then attempts to download a .tar.gz from this >>>>>>>> location: >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> = http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15= B03.tar.gz >>>>>>>>=20 >>>>>>>> It should be obvious how the URL is constructed from the POST = data. >>>>>>>> (This url is live, feel free to play around with this tarball). >>>>>>>>=20 >>>>>>>> Next it calculates the sha hash for the downloaded .tar.gz file = and >>>>>>>> matches it against the correct version in the `checksums` = parameter. >>>>>>>>=20 >>>>>>>> If that succeeds, we unpack the .tar.gz file (currently in = `/tmp`, >>>>>>>> need to find a better place for this) and adds the extracted = directory >>>>>>>> to the Erlang code path >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> = (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea= 0bea-R15B03/ebin")`) >>>>>>>> and loads the included application = (`application:load(geocouch)`). >>>>>>>>=20 >>>>>>>> Then it looks into the `./config` directory that lives next to = `ebin/` >>>>>>>> in the plugin directory for a file `config.erlt` (=93erl-terms=94= ). with a >>>>>>>> list of configuration parameters to load. We parse the file and = set >>>>>>>> the config directives one by one. >>>>>>>>=20 >>>>>>>> If that all goes to plan, we report success back to the HTTP = caller. >>>>>>>>=20 >>>>>>>> That=92s it! :) >>>>>>>>=20 >>>>>>>> It=92s deceptively simple, probably does a few things very = wrong and >>>>>>>> leaves a few things open (see above). >>>>>>>>=20 >>>>>>>> One open question I=92d like an answer for is finding a good = location to >>>>>>>> unpack & install the plugin files that isn=92t `tmp`. If the = answer is >>>>>>>> different for a pre-BigCouch/rcouch-merge and = post-BigCouch/rcouch- >>>>>>>> merge world, I=92d love to know :) >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ## Code >>>>>>>>=20 >>>>>>>> The main branch for this is 1867-feature-plugins: >>>>>>>>=20 >>>>>>>> ASF: >>>>>> = https://git-wip-us.apache.org/repos/asf?p=3Dcouchdb.git;a=3Dlog;h=3Drefs/h= eads/1867-feature-plugins >>>>>>>> GitHub: = https://github.com/janl/couchdb/compare/1867-feature-plugins >>>>>>>>=20 >>>>>>>> I created a branch on GeoCouch that adds a few lines to its = `Makefile` >>>>>>>> that shows how a binary package is built: >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> = https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1= .3.x-plugins >>>>>>>>=20 >>>>>>>> * * * >>>>>>>>=20 >>>>>>>> I hope you like this :) Please comment and improve heavily! >>>>>>>>=20 >>>>>>>> Let me know if you have any questions :) >>>>>>>>=20 >>>>>>>> If you have any criticism, please phrase it in a way that we = can use >>>>>>>> to improve this, thanks! >>>>>>>>=20 >>>>>>>> Best, >>>>>>>> Jan >>>>>>>> -- >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >=20 >=20 --Apple-Mail=_00CCB7F7-801F-4782-BC00-0577EAA7010D Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlH9ZOwACgkQ7KW8t7uWVrAtIgCZAXf5VcLSQ7UA4lyTC/Qi36y7 ciwAoMoxvOPsd36hrTgr2vcLQ9Btx+4E =NJJW -----END PGP SIGNATURE----- --Apple-Mail=_00CCB7F7-801F-4782-BC00-0577EAA7010D--