incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Minimally Viable Plugins
Date Sun, 04 Aug 2013 17:39:36 GMT


On 04.08.2013, at 18:03, Octavian Damiean <mainerror@gmail.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Oh yea, forgot to tell. Done. :)


Merged, thanks!



> 
> On 2013-08-04 17:39, Jan Lehnardt wrote:
>> 
>> On Aug 4, 2013, at 17:29 , Octavian Damiean <mainerror@gmail.com>
>> wrote:
>> 
>>> Signed PGP part Great work!
>>> 
>>> Got a quick question. In the notice you mention that the
>>> instructions work only with the 1867-feature-plugin branch but
>>> you didn't add steps to checkout said branch in the preparation
>>> step.
>>> 
>>> Want me to add that bit of information or was that deliberate?
>> 
>> Total oversight, you are more then welcome to add this :)
>> 
>> Jan --
>> 
>> 
>> 
>>> 
>>> Cheers, Octavian
>>> 
>>> On 2013-08-03 22:15, Jan Lehnardt wrote:
>>>> More update:
>>>> 
>>>> I started producing a plugin template repo that people can
>>>> clone to build their own plugins along with a comprehensive
>>>> README.md:
>>>> 
>>>> https://github.com/janl/my-first-couchdb-plugin
>>>> 
>>>> The idea is to move the README to the CouchDB docs eventually
>>>> and ship the plugin template with CouchDB, so people can get
>>>> started easily.
>>>> 
>>>> Best Jan --
>>>> 
>>>> On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com> 
>>>> wrote:
>>>> 
>>>>> :)
>>>>> 
>>>>> 
>>>>> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
>>>>> 
>>>>>> Couldn’t help but implement it. It’s in the branch now.
>>>>>> 
>>>>>> Jan --
>>>>>> 
>>>>>> On Aug 3, 2013, at 08:12 , Simon Metson
>>>>>> <simon@cloudant.com> wrote:
>>>>>> 
>>>>>>> Sounds good to me.
>>>>>>> 
>>>>>>> 
>>>>>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> On Aug 3, 2013, at 00:02 , Russell Branca 
>>>>>>>> <chewbranca@apache.org (mailto:chewbranca@apache.org)>

>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> This is fantastic, Jan! Glad to see this coming
>>>>>>>>> along.
>>>>>>>>> 
>>>>>>>>> One of the goals with Fauxton has always been to make
>>>>>>>>> it easy for plugins to extend the interface and
>>>>>>>>> provide new functionality. I've been toying with the
>>>>>>>>> idea of having a _fauxton db that plugins install to
>>>>>>>>> as docs with attachments, but that's for a different
>>>>>>>>> thread. The general idea here is that a plugin will
>>>>>>>>> be able to extend Fauxton by adding a new page with
>>>>>>>>> it's own functionality, or hook into existing pages
>>>>>>>>> to extend other areas.
>>>>>>>>> 
>>>>>>>>> For instance, you could have a couchdb-lucene plugin
>>>>>>>>> that hooks into the databases list and allows you to
>>>>>>>>> add interfaces for building full text indexes and
>>>>>>>>> searching on existing indexes. Or you could have a
>>>>>>>>> dedicated page for Geocouch, or whatever.
>>>>>>>>> 
>>>>>>>>> The functionality is there, but it's still a bit of
>>>>>>>>> a manual process, so we'll need to make it more
>>>>>>>>> dynamic and smooth out the rough edges.
>>>>>>>>> 
>>>>>>>>> I'm very excited to see progress being made on
>>>>>>>>> plugins, great work!
>>>>>>>> 
>>>>>>>> Thanks, I’m glad you like this! :)
>>>>>>>> 
>>>>>>>> Another way to get the Fauxton plugin loaded would be
>>>>>>>> to extend the /_plugins API endpoint, so Fauxton could
>>>>>>>> request GET /_plugins/<pluginname>/ and it would serve

>>>>>>>> <couchdblibdir>/plugins/<pluginname/priv/www which
is
>>>>>>>> just a place for Fauxton-enabled plugins.
>>>>>>>> 
>>>>>>>> Fauxton would walk /_config/plugins/ to get to a list
>>>>>>>> of plugins.
>>>>>>>> 
>>>>>>>> In fact that should be pretty simple to set up.
>>>>>>>> 
>>>>>>>> For now I am trying to avoid having a custom database
>>>>>>>> for this, mostly because I don’t think there are many 
>>>>>>>> advantages (e.g. replication of plugins?) and code 
>>>>>>>> complexity. These priorities might change in the
>>>>>>>> future, but for now I am happy to get this working at
>>>>>>>> all :)
>>>>>>>> 
>>>>>>>> If you are okay with the above plan of serving plugin 
>>>>>>>> HTML/JS/CSS from /_plugins/<pluginname>, I’m happy
to
>>>>>>>> add this to the branch.
>>>>>>>> 
>>>>>>>> Best Jan --
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -Russell
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt 
>>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>>> 
>>>>>>>>>> And a few more (from COUCHDB-1867):
>>>>>>>>>> 
>>>>>>>>>> - Add uninstall, incl. Futon UI. - Only install a

>>>>>>>>>> plugin if the source and target CouchDB version 
>>>>>>>>>> matches. - Rebase against master.
>>>>>>>>>> 
>>>>>>>>>> * * *
>>>>>>>>>> 
>>>>>>>>>> This concludes my list for a Minimally Viable
>>>>>>>>>> Plugin feature. (See the original email or
>>>>>>>>>> README.md (http://README.md)* for the roadmap)
>>>>>>>>>> 
>>>>>>>>>> I’d appreciate some more reviews & feedback**,
but 
>>>>>>>>>> other than that, I’d be happy to ship this as an

>>>>>>>>>> experimental feature in any next release.
>>>>>>>>>> 
>>>>>>>>>> * 
>>>>>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>>> 
> **
>>>>>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>>> 
> Best
>>>>>>>>>> Jan --
>>>>>>>>>> 
>>>>>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt
>>>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>>>> 
>>>>>>>>>>> A few updates:
>>>>>>>>>>> 
>>>>>>>>>>> By Bob Ippolito / @etrepum: - Plugins are now

>>>>>>>>>>> installed in libdir (instead of /tmp). - Config

>>>>>>>>>>> loading is now done with proper .ini files. -
>>>>>>>>>>> Various cleanups and code review (Thanks!).
>>>>>>>>>>> 
>>>>>>>>>>> Mine (most suggested by Bob): - `plugins.html`
>>>>>>>>>>> now shows you if a plugin is already installed.
>>>>>>>>>>> and which version, if it doesn’t match the
>>>>>>>>>>> installable one. - The Install button now
>>>>>>>>>>> disables after an installation. - Plugins are
now
>>>>>>>>>>> registered with couch_config as
>>>>>>>>>>> /_config/plugins/name = version - Updated
>>>>>>>>>>> `couch-config` to print --erlang-version and

>>>>>>>>>>> --erl-bin - Updated the geocouch plugin to use
>>>>>>>>>>> the new options in `couch-config`. - Added Bob
>>>>>>>>>>> Ippolito’s couchperuser plugin to Futon.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Best Jan --
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt 
>>>>>>>>>>> <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Heya,
>>>>>>>>>>>> 
>>>>>>>>>>>> I couldn’t help myself thinking about plugin
>>>>>>>>>>>> stuff and ended up whipping up a proof of
>>>>>>>>>>>> concept.
>>>>>>>>>>>> 
>>>>>>>>>>>> Here’s a <1 minute demo video:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>>> 
> Alternative encoding:
>>>>>>>>>>>> 
>>>>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>>> 
> In my head the whole plugin idea is a very wide area, but I was so
>>>>>>>>>>>> intrigued by the idea of getting something
>>>>>>>>>>>> running with a click on a button in Futon.
So I
>>>>>>>>>>>> looked for a minimally viable plugin system.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ## Design principles
>>>>>>>>>>>> 
>>>>>>>>>>>> It took me a day to put this all together
and
>>>>>>>>>>>> this was only possible because I took a lot
of 
>>>>>>>>>>>> shortcuts. I believe they are all viable
for a
>>>>>>>>>>>> first iteration of a plugins system:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Install with one click on a button in
Futon
>>>>>>>>>>>> (or HTTP call) 2. Only pure Erlang plugins
are 
>>>>>>>>>>>> allowed. 3. The plugin author must provide
a
>>>>>>>>>>>> binary package for each Erlang (and, later,
>>>>>>>>>>>> each CouchDB version). 4. Complete trust-based
>>>>>>>>>>>> system. You trust me to not do any nasty
things
>>>>>>>>>>>> when you click on the install button. No
>>>>>>>>>>>> crypto, no nothing. Only people who can commit
>>>>>>>>>>>> to Futon can release new versions of plugins.
>>>>>>>>>>>> 5. Minimal user-friendlyness: won’t install
>>>>>>>>>>>> plugins that don’t match the current Erlang

>>>>>>>>>>>> version, gives semi-sensible error messages

>>>>>>>>>>>> (wrapped in a HTTP 500 response :) 6. Require
>>>>>>>>>>>> a pretty strict format for binary releases.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ## Roadmap
>>>>>>>>>>>> 
>>>>>>>>>>>> Here’s a list of things this first iterations
>>>>>>>>>>>> does and doesn’t do:
>>>>>>>>>>>> 
>>>>>>>>>>>> - Pure Erlang plugins only. No C-dependencies,
>>>>>>>>>>>> no JavaScript, no
>>>>>>>>>> nothing.
>>>>>>>>>>>> - No C-dependencies. - Install a plugin via
>>>>>>>>>>>> Futon (or HTTP call). Admin only. - A hardcoded
>>>>>>>>>>>> list of plugins in Futon. - Loads a
>>>>>>>>>>>> pre-packaged, pre-compiled .tar.gz file from
a
>>>>>>>>>>>> URL. - Only installs if Erlang version matches.
>>>>>>>>>>>> - No security checking of binaries. - No
>>>>>>>>>>>> identity checking of binaries.
>>>>>>>>>>>> 
>>>>>>>>>>>> Here are a few things I want to add before
I
>>>>>>>>>>>> call it MVP*:
>>>>>>>>>>>> 
>>>>>>>>>>>> - Uninstall a plugin via Futon (or HTTP call).

>>>>>>>>>>>> Admin only. - Only installs if CouchDB version

>>>>>>>>>>>> matches. - Binaries must be published on

>>>>>>>>>>>> *.apache.org (http://apache.org). - Register

>>>>>>>>>>>> installed plugins in the config system. -
Make
>>>>>>>>>>>> sure plugins start with the next restart
of
>>>>>>>>>>>> CouchDB. - Show when a particular plugin
is
>>>>>>>>>>>> installed.
>>>>>>>>>>>> 
>>>>>>>>>>>> *MVP hopefully means you agree we can ship
>>>>>>>>>>>> this with a few warnings so people can get
a
>>>>>>>>>>>> hang of it.
>>>>>>>>>>>> 
>>>>>>>>>>>> Here is a rough list of features squared
>>>>>>>>>>>> against future milestones:
>>>>>>>>>>>> 
>>>>>>>>>>>> Milestone 2: Be creator friendly - Make it
easy
>>>>>>>>>>>> to build a CouchDB plugin by providing one
or
>>>>>>>>>>>> more easy to start templates. - Make it easy
to
>>>>>>>>>>>> publish new plugins and new versions of
>>>>>>>>>>>> existing
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> plugins.
>>>>>>>>>>>> - Make it easy to supply packages for multiple

>>>>>>>>>>>> Erlang & CouchDB
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> versions.
>>>>>>>>>>>> 
>>>>>>>>>>>> Milestone 3: Public registry - Instead of

>>>>>>>>>>>> hardcoding a list of plugins into
>>>>>>>>>>>> Futon/Fauxton, we load a list of applicable
>>>>>>>>>>>> plugins from a central (and configurable)
>>>>>>>>>>>> plugins repository. - This allows plugin
>>>>>>>>>>>> authors to publish new plugins and new versions
>>>>>>>>>>>> of existing plugins independently.
>>>>>>>>>>>> 
>>>>>>>>>>>> Milestone 4: Other Languages - Figure out
how
>>>>>>>>>>>> to handle C-dependencies for Erlang plugins.
-
>>>>>>>>>>>> Figure out how to allow other language plugins
>>>>>>>>>>>> (c.f. non-JS query servers)
>>>>>>>>>>>> 
>>>>>>>>>>>> Milestone X: Later - Add some 
>>>>>>>>>>>> account/identity/maybe crypto-web-of-trust
>>>>>>>>>>>> system for authors to publish “legit”
plugins.
>>>>>>>>>>>> - Sign & verify individual releases.
>>>>>>>>>>>> 
>>>>>>>>>>>> A few more things that can happen concurrently

>>>>>>>>>>>> depending on what plugins require: - Integrate

>>>>>>>>>>>> Erlang/JS tests in the installation -
>>>>>>>>>>>> Integrate docs
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ## How it works
>>>>>>>>>>>> 
>>>>>>>>>>>> This plugin system lives in `src/couch_plugins`
>>>>>>>>>>>> and is a tiny CouchDB module.
>>>>>>>>>>>> 
>>>>>>>>>>>> It exposes one new API endpoint `/_plugins`
>>>>>>>>>>>> that an admin user can POST to.
>>>>>>>>>>>> 
>>>>>>>>>>>> The additional Futon page lives at 
>>>>>>>>>>>> /_utils/plugins.html it is hardcoded.
>>>>>>>>>>>> 
>>>>>>>>>>>> Futon (or you) post an object to `/_plugins`
>>>>>>>>>>>> with four properties:
>>>>>>>>>>>> 
>>>>>>>>>>>> { "name": "geocouch", // name of the plugin,
>>>>>>>>>>>> must be unique "url":
>>>>>>>>>>>> "http://people.apache.org/~jan", // “base
URL”
>>>>>>>>>>>> for plugin
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> releases (see below)
>>>>>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22",
>>>>>>>>>>>> // whatever version
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> internal to the plugin
>>>>>>>>>>>> "checksums": { "R15B03": 
>>>>>>>>>>>> "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d
sha 
>>>>>>>>>>>> hash
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> over the binary
>>>>>>>>>>>> } }
>>>>>>>>>>>> 
>>>>>>>>>>>> `couch_plugins` then attempts to download
a
>>>>>>>>>>>> .tar.gz from this location:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>>> 
> It should be obvious how the URL is constructed from the POST data.
>>>>>>>>>>>> (This url is live, feel free to play around
>>>>>>>>>>>> with this tarball).
>>>>>>>>>>>> 
>>>>>>>>>>>> Next it calculates the sha hash for the
>>>>>>>>>>>> downloaded .tar.gz file and matches it against
>>>>>>>>>>>> the correct version in the `checksums`
>>>>>>>>>>>> parameter.
>>>>>>>>>>>> 
>>>>>>>>>>>> If that succeeds, we unpack the .tar.gz file

>>>>>>>>>>>> (currently in `/tmp`, need to find a better
>>>>>>>>>>>> place for this) and adds the extracted
>>>>>>>>>>>> directory to the Erlang code path
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>>> 
> and loads the included application (`application:load(geocouch)`).
>>>>>>>>>>>> 
>>>>>>>>>>>> Then it looks into the `./config` directory
>>>>>>>>>>>> that lives next to `ebin/` in the plugin
>>>>>>>>>>>> directory for a file `config.erlt`
>>>>>>>>>>>> (“erl-terms”). with a list of configuration
>>>>>>>>>>>> parameters to load. We parse the file and
set
>>>>>>>>>>>> the config directives one by one.
>>>>>>>>>>>> 
>>>>>>>>>>>> If that all goes to plan, we report success
>>>>>>>>>>>> back to the HTTP caller.
>>>>>>>>>>>> 
>>>>>>>>>>>> That’s it! :)
>>>>>>>>>>>> 
>>>>>>>>>>>> It’s deceptively simple, probably does
a few
>>>>>>>>>>>> things very wrong and leaves a few things
open
>>>>>>>>>>>> (see above).
>>>>>>>>>>>> 
>>>>>>>>>>>> One open question I’d like an answer for
is
>>>>>>>>>>>> finding a good location to unpack & install
the
>>>>>>>>>>>> plugin files that isn’t `tmp`. If the answer
is
>>>>>>>>>>>> different for a pre-BigCouch/rcouch-merge
and 
>>>>>>>>>>>> post-BigCouch/rcouch- merge world, I’d
love to
>>>>>>>>>>>> know :)
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> ## Code
>>>>>>>>>>>> 
>>>>>>>>>>>> The main branch for this is
>>>>>>>>>>>> 1867-feature-plugins:
>>>>>>>>>>>> 
>>>>>>>>>>>> ASF:
>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>>> 
> GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>>>>>>>>>>>> 
>>>>>>>>>>>> I created a branch on GeoCouch that adds
a few 
>>>>>>>>>>>> lines to its `Makefile` that shows how a
>>>>>>>>>>>> binary package is built:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>>> 
> * * *
>>>>>>>>>>>> 
>>>>>>>>>>>> I hope you like this :) Please comment and
>>>>>>>>>>>> improve heavily!
>>>>>>>>>>>> 
>>>>>>>>>>>> Let me know if you have any questions :)
>>>>>>>>>>>> 
>>>>>>>>>>>> If you have any criticism, please phrase
it in
>>>>>>>>>>>> a way that we can use to improve this, thanks!
>>>>>>>>>>>> 
>>>>>>>>>>>> Best, Jan --
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iQEcBAEBAgAGBQJR/ntiAAoJEAq942X94y8mpkYIAK1w9MBOmXSw3BZc9OJf55W1
> 4KRHvtvpyUy5BTZyoCHCeKquHDIy+rRJTnX69y3k4yXwEAG6kocJ5UldWUSn/tX8
> RAnEyIeTQs5YAjybutPIjpw6ClXgJnxON4sDCjvDPMoj3csE2RbxHqMD1udSGDd5
> OuUdqYBvC2g8XpNq3hbxtX7O0982s4OOfdJAywTpQ4vrcBhql+5m70YUiM+zpRF2
> JjQxngk9110aMEBDm6CgjW6z7XtwMbWqsXuFDfr5/RvFwBdAD+PksfWvPyUOmq2x
> Bga7Dqa+clBN6AmA6oNGibXcIAjL//vqynFK4XKOSBchKuLurPypE5b30pkzmX4=
> =S/C3
> -----END PGP SIGNATURE-----

Mime
View raw message