Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of mainerror@gmail.com
 designates 74.125.82.179 as permitted sender)
Message-ID: <51FE733F.9050600@gmail.com>
Date: Sun, 04 Aug 2013 17:29:03 +0200
From: Octavian Damiean <mainerror@gmail.com>
Reply-To: odamiean@linux.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130623 Thunderbird/17.0.7
MIME-Version: 1.0
To: dev@couchdb.apache.org
Subject: Re: Minimally Viable Plugins
References: <9BF26E7B-D6D4-4D1F-A11B-771038B7FACB@apache.org>
 <6955D445-3D8A-482B-8F4F-12F94B1B12D0@apache.org>
 <3A847F88-3E70-4463-9E91-383290D871F5@apache.org>
 <CAFUSCh8PB6rhbXDd8z7eNfSe0+NJZLu5xiZqBQwRQDnKfePA1Q@mail.gmail.com>
 <E385DDAF-49D6-4322-80AC-17E6D4E41AFC@apache.org>
 <A857685D055142C983C763586C27FD4C@cloudant.com>
 <E5C360AD-2ED2-43ED-B82B-90AD2A906394@apache.org>
 <3754CC5CF2D84A32B807CB0E681AFDE4@cloudant.com>
 <20E0B52F-5B43-4E16-AAB6-3DA495F6261B@apache.org>
In-Reply-To: <20E0B52F-5B43-4E16-AAB6-3DA495F6261B@apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Great work!

Got a quick question. In the notice you mention that the instructions
work only with the 1867-feature-plugin branch but you didn't add steps
to checkout said branch in the preparation step.

Want me to add that bit of information or was that deliberate?

Cheers,
Octavian

On 2013-08-03 22:15, Jan Lehnardt wrote:
> More update:
> 
> I started producing a plugin template repo that people can clone
> to build their own plugins along with a comprehensive README.md:
> 
> https://github.com/janl/my-first-couchdb-plugin
> 
> The idea is to move the README to the CouchDB docs eventually and
> ship the plugin template with CouchDB, so people can get started
> easily.
> 
> Best Jan --
> 
> On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com>
> wrote:
> 
>> :)
>> 
>> 
>> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
>> 
>>> Couldn�t help but implement it. It�s in the branch now.
>>> 
>>> Jan --
>>> 
>>> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com>
>>> wrote:
>>> 
>>>> Sounds good to me.
>>>> 
>>>> 
>>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
>>>> 
>>>>> 
>>>>> On Aug 3, 2013, at 00:02 , Russell Branca
>>>>> <chewbranca@apache.org (mailto:chewbranca@apache.org)>
>>>>> wrote:
>>>>> 
>>>>>> This is fantastic, Jan! Glad to see this coming along.
>>>>>> 
>>>>>> One of the goals with Fauxton has always been to make it
>>>>>> easy for plugins to extend the interface and provide new
>>>>>> functionality. I've been toying with the idea of having a
>>>>>> _fauxton db that plugins install to as docs with 
>>>>>> attachments, but that's for a different thread. The
>>>>>> general idea here is that a plugin will be able to extend
>>>>>> Fauxton by adding a new page with it's own functionality,
>>>>>> or hook into existing pages to extend other areas.
>>>>>> 
>>>>>> For instance, you could have a couchdb-lucene plugin that
>>>>>> hooks into the databases list and allows you to add
>>>>>> interfaces for building full text indexes and searching
>>>>>> on existing indexes. Or you could have a dedicated page
>>>>>> for Geocouch, or whatever.
>>>>>> 
>>>>>> The functionality is there, but it's still a bit of a
>>>>>> manual process, so we'll need to make it more dynamic and
>>>>>> smooth out the rough edges.
>>>>>> 
>>>>>> I'm very excited to see progress being made on plugins,
>>>>>> great work!
>>>>> 
>>>>> Thanks, I�m glad you like this! :)
>>>>> 
>>>>> Another way to get the Fauxton plugin loaded would be to
>>>>> extend the /_plugins API endpoint, so Fauxton could request
>>>>> GET /_plugins/<pluginname>/ and it would serve
>>>>> <couchdblibdir>/plugins/<pluginname/priv/www which is just
>>>>> a place for Fauxton-enabled plugins.
>>>>> 
>>>>> Fauxton would walk /_config/plugins/ to get to a list of
>>>>> plugins.
>>>>> 
>>>>> In fact that should be pretty simple to set up.
>>>>> 
>>>>> For now I am trying to avoid having a custom database for
>>>>> this, mostly because I don�t think there are many
>>>>> advantages (e.g. replication of plugins?) and code
>>>>> complexity. These priorities might change in the future,
>>>>> but for now I am happy to get this working at all :)
>>>>> 
>>>>> If you are okay with the above plan of serving plugin
>>>>> HTML/JS/CSS from /_plugins/<pluginname>, I�m happy to add
>>>>> this to the branch.
>>>>> 
>>>>> Best Jan --
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -Russell
>>>>>> 
>>>>>> 
>>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt
>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>> 
>>>>>>> And a few more (from COUCHDB-1867):
>>>>>>> 
>>>>>>> - Add uninstall, incl. Futon UI. - Only install a
>>>>>>> plugin if the source and target CouchDB version
>>>>>>> matches. - Rebase against master.
>>>>>>> 
>>>>>>> * * *
>>>>>>> 
>>>>>>> This concludes my list for a Minimally Viable Plugin
>>>>>>> feature. (See the original email or README.md
>>>>>>> (http://README.md)* for the roadmap)
>>>>>>> 
>>>>>>> I�d appreciate some more reviews & feedback**, but
>>>>>>> other than that, I�d be happy to ship this as an
>>>>>>> experimental feature in any next release.
>>>>>>> 
>>>>>>> * 
>>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>>>>>>>
>>>>>>> 
**
>>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 
Best
>>>>>>> Jan --
>>>>>>> 
>>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org
>>>>>>> (mailto:jan@apache.org)> wrote:
>>>>>>> 
>>>>>>>> A few updates:
>>>>>>>> 
>>>>>>>> By Bob Ippolito / @etrepum: - Plugins are now
>>>>>>>> installed in libdir (instead of /tmp). - Config
>>>>>>>> loading is now done with proper .ini files. - Various
>>>>>>>> cleanups and code review (Thanks!).
>>>>>>>> 
>>>>>>>> Mine (most suggested by Bob): - `plugins.html` now
>>>>>>>> shows you if a plugin is already installed. and which
>>>>>>>> version, if it doesn�t match the installable one. -
>>>>>>>> The Install button now disables after an
>>>>>>>> installation. - Plugins are now registered with
>>>>>>>> couch_config as /_config/plugins/name = version -
>>>>>>>> Updated `couch-config` to print --erlang-version and
>>>>>>>> --erl-bin - Updated the geocouch plugin to use the
>>>>>>>> new options in `couch-config`. - Added Bob Ippolito�s
>>>>>>>> couchperuser plugin to Futon.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Best Jan --
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt
>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>> 
>>>>>>>>> Heya,
>>>>>>>>> 
>>>>>>>>> I couldn�t help myself thinking about plugin stuff
>>>>>>>>> and ended up whipping up a proof of concept.
>>>>>>>>> 
>>>>>>>>> Here�s a <1 minute demo video:
>>>>>>>>> 
>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 
Alternative encoding:
>>>>>>>>> 
>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 
In my head the whole plugin idea is a very wide area, but I was so
>>>>>>>>> intrigued by the idea of getting something running
>>>>>>>>> with a click on a button in Futon. So I looked for
>>>>>>>>> a minimally viable plugin system.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ## Design principles
>>>>>>>>> 
>>>>>>>>> It took me a day to put this all together and this
>>>>>>>>> was only possible because I took a lot of
>>>>>>>>> shortcuts. I believe they are all viable for a 
>>>>>>>>> first iteration of a plugins system:
>>>>>>>>> 
>>>>>>>>> 1. Install with one click on a button in Futon (or
>>>>>>>>> HTTP call) 2. Only pure Erlang plugins are
>>>>>>>>> allowed. 3. The plugin author must provide a binary
>>>>>>>>> package for each Erlang (and, later, each CouchDB
>>>>>>>>> version). 4. Complete trust-based system. You trust
>>>>>>>>> me to not do any nasty things when you click on the
>>>>>>>>> install button. No crypto, no nothing. Only people
>>>>>>>>> who can commit to Futon can release new versions of
>>>>>>>>> plugins. 5. Minimal user-friendlyness: won�t
>>>>>>>>> install plugins that don�t match the current Erlang
>>>>>>>>> version, gives semi-sensible error messages 
>>>>>>>>> (wrapped in a HTTP 500 response :) 6. Require a
>>>>>>>>> pretty strict format for binary releases.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ## Roadmap
>>>>>>>>> 
>>>>>>>>> Here�s a list of things this first iterations does
>>>>>>>>> and doesn�t do:
>>>>>>>>> 
>>>>>>>>> - Pure Erlang plugins only. No C-dependencies, no
>>>>>>>>> JavaScript, no
>>>>>>> nothing.
>>>>>>>>> - No C-dependencies. - Install a plugin via Futon
>>>>>>>>> (or HTTP call). Admin only. - A hardcoded list of
>>>>>>>>> plugins in Futon. - Loads a pre-packaged,
>>>>>>>>> pre-compiled .tar.gz file from a URL. - Only
>>>>>>>>> installs if Erlang version matches. - No security
>>>>>>>>> checking of binaries. - No identity checking of
>>>>>>>>> binaries.
>>>>>>>>> 
>>>>>>>>> Here are a few things I want to add before I call
>>>>>>>>> it MVP*:
>>>>>>>>> 
>>>>>>>>> - Uninstall a plugin via Futon (or HTTP call).
>>>>>>>>> Admin only. - Only installs if CouchDB version
>>>>>>>>> matches. - Binaries must be published on
>>>>>>>>> *.apache.org (http://apache.org). - Register
>>>>>>>>> installed plugins in the config system. - Make sure
>>>>>>>>> plugins start with the next restart of CouchDB. -
>>>>>>>>> Show when a particular plugin is installed.
>>>>>>>>> 
>>>>>>>>> *MVP hopefully means you agree we can ship this
>>>>>>>>> with a few warnings so people can get a hang of
>>>>>>>>> it.
>>>>>>>>> 
>>>>>>>>> Here is a rough list of features squared against
>>>>>>>>> future milestones:
>>>>>>>>> 
>>>>>>>>> Milestone 2: Be creator friendly - Make it easy to
>>>>>>>>> build a CouchDB plugin by providing one or more
>>>>>>>>> easy to start templates. - Make it easy to publish
>>>>>>>>> new plugins and new versions of existing
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> plugins.
>>>>>>>>> - Make it easy to supply packages for multiple
>>>>>>>>> Erlang & CouchDB
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> versions.
>>>>>>>>> 
>>>>>>>>> Milestone 3: Public registry - Instead of
>>>>>>>>> hardcoding a list of plugins into Futon/Fauxton, we
>>>>>>>>> load a list of applicable plugins from a central
>>>>>>>>> (and configurable) plugins repository. - This
>>>>>>>>> allows plugin authors to publish new plugins and
>>>>>>>>> new versions of existing plugins independently.
>>>>>>>>> 
>>>>>>>>> Milestone 4: Other Languages - Figure out how to
>>>>>>>>> handle C-dependencies for Erlang plugins. - Figure
>>>>>>>>> out how to allow other language plugins (c.f.
>>>>>>>>> non-JS query servers)
>>>>>>>>> 
>>>>>>>>> Milestone X: Later - Add some
>>>>>>>>> account/identity/maybe crypto-web-of-trust system
>>>>>>>>> for authors to publish �legit� plugins. - Sign &
>>>>>>>>> verify individual releases.
>>>>>>>>> 
>>>>>>>>> A few more things that can happen concurrently
>>>>>>>>> depending on what plugins require: - Integrate
>>>>>>>>> Erlang/JS tests in the installation - Integrate
>>>>>>>>> docs
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ## How it works
>>>>>>>>> 
>>>>>>>>> This plugin system lives in `src/couch_plugins` and
>>>>>>>>> is a tiny CouchDB module.
>>>>>>>>> 
>>>>>>>>> It exposes one new API endpoint `/_plugins` that an
>>>>>>>>> admin user can POST to.
>>>>>>>>> 
>>>>>>>>> The additional Futon page lives at
>>>>>>>>> /_utils/plugins.html it is hardcoded.
>>>>>>>>> 
>>>>>>>>> Futon (or you) post an object to `/_plugins` with
>>>>>>>>> four properties:
>>>>>>>>> 
>>>>>>>>> { "name": "geocouch", // name of the plugin, must
>>>>>>>>> be unique "url": "http://people.apache.org/~jan",
>>>>>>>>> // �base URL� for plugin
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> releases (see below)
>>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", //
>>>>>>>>> whatever version
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> internal to the plugin
>>>>>>>>> "checksums": { "R15B03":
>>>>>>>>> "ZetgdHj2bY2w37buulWVf3USOZs=" // base64�d sha
>>>>>>>>> hash
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> over the binary
>>>>>>>>> } }
>>>>>>>>> 
>>>>>>>>> `couch_plugins` then attempts to download a .tar.gz
>>>>>>>>> from this location:
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>>>>>>>>>
>>>>>>>>>
>>>>>>> 
It should be obvious how the URL is constructed from the POST data.
>>>>>>>>> (This url is live, feel free to play around with
>>>>>>>>> this tarball).
>>>>>>>>> 
>>>>>>>>> Next it calculates the sha hash for the downloaded
>>>>>>>>> .tar.gz file and matches it against the correct
>>>>>>>>> version in the `checksums` parameter.
>>>>>>>>> 
>>>>>>>>> If that succeeds, we unpack the .tar.gz file
>>>>>>>>> (currently in `/tmp`, need to find a better place
>>>>>>>>> for this) and adds the extracted directory to the
>>>>>>>>> Erlang code path
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>>>>>>>>>
>>>>>>> 
and loads the included application (`application:load(geocouch)`).
>>>>>>>>> 
>>>>>>>>> Then it looks into the `./config` directory that
>>>>>>>>> lives next to `ebin/` in the plugin directory for a
>>>>>>>>> file `config.erlt` (�erl-terms�). with a list of
>>>>>>>>> configuration parameters to load. We parse the file
>>>>>>>>> and set the config directives one by one.
>>>>>>>>> 
>>>>>>>>> If that all goes to plan, we report success back to
>>>>>>>>> the HTTP caller.
>>>>>>>>> 
>>>>>>>>> That�s it! :)
>>>>>>>>> 
>>>>>>>>> It�s deceptively simple, probably does a few things
>>>>>>>>> very wrong and leaves a few things open (see
>>>>>>>>> above).
>>>>>>>>> 
>>>>>>>>> One open question I�d like an answer for is finding
>>>>>>>>> a good location to unpack & install the plugin
>>>>>>>>> files that isn�t `tmp`. If the answer is different
>>>>>>>>> for a pre-BigCouch/rcouch-merge and
>>>>>>>>> post-BigCouch/rcouch- merge world, I�d love to know
>>>>>>>>> :)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ## Code
>>>>>>>>> 
>>>>>>>>> The main branch for this is 1867-feature-plugins:
>>>>>>>>> 
>>>>>>>>> ASF:
>>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>>>>>>>>>
>>>>>>> 
GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>>>>>>>>> 
>>>>>>>>> I created a branch on GeoCouch that adds a few
>>>>>>>>> lines to its `Makefile` that shows how a binary
>>>>>>>>> package is built:
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>>>>>>>>>
>>>>>>>>>
>>>>>>> 
* * *
>>>>>>>>> 
>>>>>>>>> I hope you like this :) Please comment and improve
>>>>>>>>> heavily!
>>>>>>>>> 
>>>>>>>>> Let me know if you have any questions :)
>>>>>>>>> 
>>>>>>>>> If you have any criticism, please phrase it in a
>>>>>>>>> way that we can use to improve this, thanks!
>>>>>>>>> 
>>>>>>>>> Best, Jan --
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJR/nM/AAoJEAq942X94y8mm4kH/izU8QN+Ybw1AeeXMler3JQi
56S/tQWQUieY1T/7zHsAYYOy2qxOigmcCMcCi6E0naX24l2rUB3YDbG3I55jX6Fu
vgrzr0gbExQT/R/pRlu92fhpXHyy6Z9aeCu6evOgnjeKMMaP8iwvfVqdZWE3W9du
T06mqd+kEiSQI0+ACMWzabAXKtAHUt8/mq/6Farfuc1DRPHZWbEiYskDdciie1SM
WtvpDL/R0DCLvdRzeqE4aZZs47TtClQIJLt6QGBggsInTNSXef8ajKKDxfx0GRXP
mjrFcA7ZAEJ0UGbBmPsOMy4MmXmwTTJzHS+1HfCkiLLHGCiRQ+hLvQbo1aQXmII=
=jw2p
-----END PGP SIGNATURE-----