Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of simon@cloudant.com designates
 74.125.82.171 as permitted sender)
Date: Sat, 3 Aug 2013 16:53:23 +0100
From: Simon Metson <simon@cloudant.com>
To: dev@couchdb.apache.org
Message-ID: <3754CC5CF2D84A32B807CB0E681AFDE4@cloudant.com>
In-Reply-To: <E5C360AD-2ED2-43ED-B82B-90AD2A906394@apache.org>
References: <9BF26E7B-D6D4-4D1F-A11B-771038B7FACB@apache.org>
 <6955D445-3D8A-482B-8F4F-12F94B1B12D0@apache.org>
 <3A847F88-3E70-4463-9E91-383290D871F5@apache.org>
 <CAFUSCh8PB6rhbXDd8z7eNfSe0+NJZLu5xiZqBQwRQDnKfePA1Q@mail.gmail.com>
 <E385DDAF-49D6-4322-80AC-17E6D4E41AFC@apache.org>
 <A857685D055142C983C763586C27FD4C@cloudant.com>
 <E5C360AD-2ED2-43ED-B82B-90AD2A906394@apache.org>
Subject: Re: Minimally Viable Plugins
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="51fd2773_7ab49daf_1948"

--51fd2773_7ab49daf_1948
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

:) =20


On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:

> Couldn=E2=80=99t help but implement it. It=E2=80=99s in the branch now.=

> =20
> Jan
> -- =20
> =20
> On Aug 3, 2013, at 08:12 , Simon Metson <simon=40cloudant.com> wrote:
> =20
> > Sounds good to me. =20
> > =20
> > =20
> > On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
> > =20
> > > =20
> > > On Aug 3, 2013, at 00:02 , Russell Branca <chewbranca=40apache.org =
(mailto:chewbranca=40apache.org)> wrote:
> > > =20
> > > > This is fantastic, Jan=21 Glad to see this coming along.
> > > > =20
> > > > One of the goals with =46auxton has always been to make it easy f=
or plugins
> > > > to extend the interface and provide new functionality. I've been =
toying
> > > > with the idea of having a =5Ffauxton db that plugins install to a=
s docs with
> > > > attachments, but that's for a different thread. The general idea =
here is
> > > > that a plugin will be able to extend =46auxton by adding a new pa=
ge with it's
> > > > own functionality, or hook into existing pages to extend other ar=
eas.
> > > > =20
> > > > =46or instance, you could have a couchdb-lucene plugin that hooks=
 into the
> > > > databases list and allows you to add interfaces for building full=
 text
> > > > indexes and searching on existing indexes. Or you could have a de=
dicated
> > > > page for Geocouch, or whatever.
> > > > =20
> > > > The functionality is there, but it's still a bit of a manual proc=
ess, so
> > > > we'll need to make it more dynamic and smooth out the rough edges=
.
> > > > =20
> > > > I'm very excited to see progress being made on plugins, great wor=
k=21
> > > =20
> > > Thanks, I=E2=80=99m glad you like this=21 :)
> > > =20
> > > Another way to get the =46auxton plugin loaded would be to extend t=
he
> > > /=5Fplugins API endpoint, so =46auxton could request GET /=5Fplugin=
s/<pluginname>/
> > > and it would serve <couchdblibdir>/plugins/<pluginname/priv/www whi=
ch is
> > > just a place for =46auxton-enabled plugins.
> > > =20
> > > =46auxton would walk /=5Fconfig/plugins/ to get to a list of plugin=
s.
> > > =20
> > > In fact that should be pretty simple to set up.
> > > =20
> > > =46or now I am trying to avoid having a custom database for this, m=
ostly
> > > because I don=E2=80=99t think there are many advantages (e.g. repli=
cation of
> > > plugins=3F) and code complexity. These priorities might change in t=
he
> > > future, but for now I am happy to get this working at all :)
> > > =20
> > > If you are okay with the above plan of serving plugin HTML/JS/CSS f=
rom
> > > /=5Fplugins/<pluginname>, I=E2=80=99m happy to add this to the bran=
ch.
> > > =20
> > > Best
> > > Jan
> > > -- =20
> > > =20
> > > =20
> > > =20
> > > > =20
> > > > =20
> > > > -Russell
> > > > =20
> > > > =20
> > > > On =46ri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt <jan=40apache.org =
(mailto:jan=40apache.org)> wrote:
> > > > =20
> > > > > And a few more (from COUCHDB-1867):
> > > > > =20
> > > > > - Add uninstall, incl. =46uton UI.
> > > > > - Only install a plugin if the source and target CouchDB versio=
n matches.
> > > > > - Rebase against master.
> > > > > =20
> > > > > * * *
> > > > > =20
> > > > > This concludes my list for a Minimally Viable Plugin feature. (=
See the
> > > > > original email or README.md (http://README.md)* for the roadmap=
)
> > > > > =20
> > > > > I=E2=80=99d appreciate some more reviews & feedback**, but othe=
r than that, I=E2=80=99d be
> > > > > happy to ship this as an experimental feature in any next relea=
se.
> > > > > =20
> > > > > *
> > > > > https://github.com/janl/couchdb/blob/1867-feature-plugins/src/c=
ouch=5Fplugins/README.md=23roadmap
> > > > > **
> > > > > https://github.com/janl/couchdb/compare/apache:master...1867-fe=
ature-plugins
> > > > > =20
> > > > > =20
> > > > > Best
> > > > > Jan
> > > > > --
> > > > > =20
> > > > > On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan=40apache.org (mail=
to:jan=40apache.org)> wrote:
> > > > > =20
> > > > > > A few updates:
> > > > > > =20
> > > > > > By Bob Ippolito / =40etrepum:
> > > > > > - Plugins are now installed in libdir (instead of /tmp).
> > > > > > - Config loading is now done with proper .ini files.
> > > > > > - Various cleanups and code review (Thanks=21).
> > > > > > =20
> > > > > > Mine (most suggested by Bob):
> > > > > > - =60plugins.html=60 now shows you if a plugin is already ins=
talled.
> > > > > > and which version, if it doesn=E2=80=99t match the installabl=
e one.
> > > > > > - The Install button now disables after an installation.
> > > > > > - Plugins are now registered with couch=5Fconfig as
> > > > > > /=5Fconfig/plugins/name =3D version
> > > > > > - Updated =60couch-config=60 to print --erlang-version and --=
erl-bin
> > > > > > - Updated the geocouch plugin to use the new options in
> > > > > > =60couch-config=60.
> > > > > > - Added Bob Ippolito=E2=80=99s couchperuser plugin to =46uton=
.
> > > > > > =20
> > > > > > =20
> > > > > > Best
> > > > > > Jan
> > > > > > --
> > > > > > =20
> > > > > > =20
> > > > > > =20
> > > > > > On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan=40apache.org (m=
ailto:jan=40apache.org)> wrote:
> > > > > > =20
> > > > > > > Heya,
> > > > > > > =20
> > > > > > > I couldn=E2=80=99t help myself thinking about plugin stuff =
and ended up
> > > > > > > whipping up a proof of concept.
> > > > > > > =20
> > > > > > > Here=E2=80=99s a <1 minute demo video:
> > > > > > > =20
> > > > > > > https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-d=
emo.mov
> > > > > > > =20
> > > > > > > Alternative encoding:
> > > > > > > =20
> > > > > > > https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-d=
emo.m4v)
> > > > > > > =20
> > > > > > > =20
> > > > > > > In my head the whole plugin idea is a very wide area, but I=
 was so
> > > > > > > intrigued by the idea of getting something running with a c=
lick on a
> > > > > > > button in =46uton. So I looked for a minimally viable plugi=
n system.
> > > > > > > =20
> > > > > > > =20
> > > > > > > =23=23 Design principles
> > > > > > > =20
> > > > > > > It took me a day to put this all together and this was only=
 possible
> > > > > > > because I took a lot of shortcuts. I believe they are all v=
iable for a
> > > > > > > first iteration of a plugins system:
> > > > > > > =20
> > > > > > > 1. Install with one click on a button in =46uton (or HTTP c=
all)
> > > > > > > 2. Only pure Erlang plugins are allowed.
> > > > > > > 3. The plugin author must provide a binary package for each=
 Erlang (and,
> > > > > > > later, each CouchDB version).
> > > > > > > 4. Complete trust-based system. You trust me to not do any =
nasty things
> > > > > > > when you click on the install button. No crypto, no nothing=
. Only
> > > > > > > people who can commit to =46uton can release new versions o=
f plugins.
> > > > > > > 5. Minimal user-friendlyness: won=E2=80=99t install plugins=
 that don=E2=80=99t match
> > > > > > > the current Erlang version, gives semi-sensible error messa=
ges
> > > > > > > (wrapped in a HTTP 500 response :)
> > > > > > > 6. Require a pretty strict format for binary releases.
> > > > > > > =20
> > > > > > > =20
> > > > > > > =23=23 Roadmap
> > > > > > > =20
> > > > > > > Here=E2=80=99s a list of things this first iterations does =
and doesn=E2=80=99t do:
> > > > > > > =20
> > > > > > > - Pure Erlang plugins only. No C-dependencies, no JavaScrip=
t, no
> > > > > nothing.
> > > > > > > - No C-dependencies.
> > > > > > > - Install a plugin via =46uton (or HTTP call). Admin only.
> > > > > > > - A hardcoded list of plugins in =46uton.
> > > > > > > - Loads a pre-packaged, pre-compiled .tar.gz file from a UR=
L.
> > > > > > > - Only installs if Erlang version matches.
> > > > > > > - No security checking of binaries.
> > > > > > > - No identity checking of binaries.
> > > > > > > =20
> > > > > > > Here are a few things I want to add before I call it MVP*:
> > > > > > > =20
> > > > > > > - Uninstall a plugin via =46uton (or HTTP call). Admin only=
.
> > > > > > > - Only installs if CouchDB version matches.
> > > > > > > - Binaries must be published on *.apache.org (http://apache=
.org).
> > > > > > > - Register installed plugins in the config system.
> > > > > > > - Make sure plugins start with the next restart of CouchDB.=

> > > > > > > - Show when a particular plugin is installed.
> > > > > > > =20
> > > > > > > *MVP hopefully means you agree we can ship this with a few =
warnings
> > > > > > > so people can get a hang of it.
> > > > > > > =20
> > > > > > > Here is a rough list of features squared against future mil=
estones:
> > > > > > > =20
> > > > > > > Milestone 2: Be creator friendly
> > > > > > > - Make it easy to build a CouchDB plugin by providing one o=
r more easy
> > > > > > > to start templates.
> > > > > > > - Make it easy to publish new plugins and new versions of e=
xisting
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > plugins.
> > > > > > > - Make it easy to supply packages for multiple Erlang & Cou=
chDB
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > versions.
> > > > > > > =20
> > > > > > > Milestone 3: Public registry
> > > > > > > - Instead of hardcoding a list of plugins into =46uton/=46a=
uxton, we load
> > > > > > > a list of applicable plugins from a central (and configurab=
le)
> > > > > > > plugins repository.
> > > > > > > - This allows plugin authors to publish new plugins and new=
 versions
> > > > > > > of existing plugins independently.
> > > > > > > =20
> > > > > > > Milestone 4: Other Languages
> > > > > > > - =46igure out how to handle C-dependencies for Erlang plug=
ins.
> > > > > > > - =46igure out how to allow other language plugins
> > > > > > > (c.f. non-JS query servers)
> > > > > > > =20
> > > > > > > Milestone X: Later
> > > > > > > - Add some account/identity/maybe crypto-web-of-trust syste=
m for
> > > > > > > authors to publish =E2=80=9Clegit=E2=80=9D plugins.
> > > > > > > - Sign & verify individual releases.
> > > > > > > =20
> > > > > > > A few more things that can happen concurrently depending on=
 what
> > > > > > > plugins require:
> > > > > > > - Integrate Erlang/JS tests in the installation
> > > > > > > - Integrate docs
> > > > > > > =20
> > > > > > > =20
> > > > > > > =23=23 How it works
> > > > > > > =20
> > > > > > > This plugin system lives in =60src/couch=5Fplugins=60 and i=
s a tiny CouchDB
> > > > > > > module.
> > > > > > > =20
> > > > > > > It exposes one new API endpoint =60/=5Fplugins=60 that an a=
dmin user can
> > > > > > > POST to.
> > > > > > > =20
> > > > > > > The additional =46uton page lives at /=5Futils/plugins.html=
 it is
> > > > > > > hardcoded.
> > > > > > > =20
> > > > > > > =46uton (or you) post an object to =60/=5Fplugins=60 with f=
our properties:
> > > > > > > =20
> > > > > > > =7B
> > > > > > > =22name=22: =22geocouch=22, // name of the plugin, must be =
unique
> > > > > > > =22url=22: =22http://people.apache.org/=7Ejan=22, // =E2=80=
=9Cbase URL=E2=80=9D for plugin
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > releases (see below)
> > > > > > > =22version=22: =22couchdb1.2.x=5Fv0.3.0-11-gd83ba22=22, // =
whatever version
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > internal to the plugin
> > > > > > > =22checksums=22: =7B
> > > > > > > =22R15B03=22: =22ZetgdHj2bY2w37buulWVf3USOZs=3D=22 // base6=
4=E2=80=99d sha hash
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > over the binary
> > > > > > > =7D
> > > > > > > =7D
> > > > > > > =20
> > > > > > > =60couch=5Fplugins=60 then attempts to download a .tar.gz f=
rom this
> > > > > > > location:
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > http://people.apache.org/=7Ejan/geocouch-couchdb1.2.x=5Fv0.3.0-=
12-g4ea0bea-R15B03.tar.gz
> > > > > > > =20
> > > > > > > It should be obvious how the URL is constructed from the PO=
ST data.
> > > > > > > (This url is live, feel free to play around with this tarba=
ll).
> > > > > > > =20
> > > > > > > Next it calculates the sha hash for the downloaded .tar.gz =
file and
> > > > > > > matches it against the correct version in the =60checksums=60=
 parameter.
> > > > > > > =20
> > > > > > > If that succeeds, we unpack the .tar.gz file (currently in =
=60/tmp=60,
> > > > > > > need to find a better place for this) and adds the extracte=
d directory
> > > > > > > to the Erlang code path
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > (=60code:add=5Fpath(=22/tmp/couchdb=5Fplugins/geocouch-couchdb1=
.2.x=5Fv0.3.0-12-g4ea0bea-R15B03/ebin=22)=60)
> > > > > > > and loads the included application (=60application:load(geo=
couch)=60).
> > > > > > > =20
> > > > > > > Then it looks into the =60./config=60 directory that lives =
next to =60ebin/=60
> > > > > > > in the plugin directory for a file =60config.erlt=60 (=E2=80=
=9Cerl-terms=E2=80=9D). with a
> > > > > > > list of configuration parameters to load. We parse the file=
 and set
> > > > > > > the config directives one by one.
> > > > > > > =20
> > > > > > > If that all goes to plan, we report success back to the HTT=
P caller.
> > > > > > > =20
> > > > > > > That=E2=80=99s it=21 :)
> > > > > > > =20
> > > > > > > It=E2=80=99s deceptively simple, probably does a few things=
 very wrong and
> > > > > > > leaves a few things open (see above).
> > > > > > > =20
> > > > > > > One open question I=E2=80=99d like an answer for is finding=
 a good location to
> > > > > > > unpack & install the plugin files that isn=E2=80=99t =60tmp=
=60. If the answer is
> > > > > > > different for a pre-BigCouch/rcouch-merge and post-BigCouch=
/rcouch-
> > > > > > > merge world, I=E2=80=99d love to know :)
> > > > > > > =20
> > > > > > > =20
> > > > > > > =23=23 Code
> > > > > > > =20
> > > > > > > The main branch for this is 1867-feature-plugins:
> > > > > > > =20
> > > > > > > AS=46:
> > > > > https://git-wip-us.apache.org/repos/asf=3Fp=3Dcouchdb.git;a=3Dl=
og;h=3Drefs/heads/1867-feature-plugins
> > > > > > > GitHub: https://github.com/janl/couchdb/compare/1867-featur=
e-plugins
> > > > > > > =20
> > > > > > > I created a branch on GeoCouch that adds a few lines to its=
 =60Makefile=60
> > > > > > > that shows how a binary package is built:
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > > =20
> > > > > https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x=
...couchdb1.3.x-plugins
> > > > > > > =20
> > > > > > > * * *
> > > > > > > =20
> > > > > > > I hope you like this :) Please comment and improve heavily=21=

> > > > > > > =20
> > > > > > > Let me know if you have any questions :)
> > > > > > > =20
> > > > > > > If you have any criticism, please phrase it in a way that w=
e can use
> > > > > > > to improve this, thanks=21
> > > > > > > =20
> > > > > > > Best,
> > > > > > > Jan
> > > > > > > --
> > > > > > > =20
> > > > > > =20
> > > > > =20
> > > > > =20
> > > > =20
> > > > =20
> > > =20
> > > =20
> > =20
> > =20
> =20
> =20
> =20


--51fd2773_7ab49daf_1948--