incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Minimally Viable Plugins
Date Sat, 03 Aug 2013 20:15:37 GMT
More update:

I started producing a plugin template repo that people can clone to
build their own plugins along with a comprehensive README.md:

    https://github.com/janl/my-first-couchdb-plugin

The idea is to move the README to the CouchDB docs eventually and ship
the plugin template with CouchDB, so people can get started easily.

Best
Jan
--

On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com> wrote:

> :)  
> 
> 
> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
> 
>> Couldn’t help but implement it. It’s in the branch now.
>> 
>> Jan
>> --  
>> 
>> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com> wrote:
>> 
>>> Sounds good to me.  
>>> 
>>> 
>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
>>> 
>>>> 
>>>> On Aug 3, 2013, at 00:02 , Russell Branca <chewbranca@apache.org (mailto:chewbranca@apache.org)>
wrote:
>>>> 
>>>>> This is fantastic, Jan! Glad to see this coming along.
>>>>> 
>>>>> One of the goals with Fauxton has always been to make it easy for plugins
>>>>> to extend the interface and provide new functionality. I've been toying
>>>>> with the idea of having a _fauxton db that plugins install to as docs
with
>>>>> attachments, but that's for a different thread. The general idea here
is
>>>>> that a plugin will be able to extend Fauxton by adding a new page with
it's
>>>>> own functionality, or hook into existing pages to extend other areas.
>>>>> 
>>>>> For instance, you could have a couchdb-lucene plugin that hooks into
the
>>>>> databases list and allows you to add interfaces for building full text
>>>>> indexes and searching on existing indexes. Or you could have a dedicated
>>>>> page for Geocouch, or whatever.
>>>>> 
>>>>> The functionality is there, but it's still a bit of a manual process,
so
>>>>> we'll need to make it more dynamic and smooth out the rough edges.
>>>>> 
>>>>> I'm very excited to see progress being made on plugins, great work!
>>>> 
>>>> Thanks, I’m glad you like this! :)
>>>> 
>>>> Another way to get the Fauxton plugin loaded would be to extend the
>>>> /_plugins API endpoint, so Fauxton could request GET /_plugins/<pluginname>/
>>>> and it would serve <couchdblibdir>/plugins/<pluginname/priv/www
which is
>>>> just a place for Fauxton-enabled plugins.
>>>> 
>>>> Fauxton would walk /_config/plugins/ to get to a list of plugins.
>>>> 
>>>> In fact that should be pretty simple to set up.
>>>> 
>>>> For now I am trying to avoid having a custom database for this, mostly
>>>> because I don’t think there are many advantages (e.g. replication of
>>>> plugins?) and code complexity. These priorities might change in the
>>>> future, but for now I am happy to get this working at all :)
>>>> 
>>>> If you are okay with the above plan of serving plugin HTML/JS/CSS from
>>>> /_plugins/<pluginname>, I’m happy to add this to the branch.
>>>> 
>>>> Best
>>>> Jan
>>>> --  
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> -Russell
>>>>> 
>>>>> 
>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>>> 
>>>>>> And a few more (from COUCHDB-1867):
>>>>>> 
>>>>>> - Add uninstall, incl. Futon UI.
>>>>>> - Only install a plugin if the source and target CouchDB version
matches.
>>>>>> - Rebase against master.
>>>>>> 
>>>>>> * * *
>>>>>> 
>>>>>> This concludes my list for a Minimally Viable Plugin feature. (See
the
>>>>>> original email or README.md (http://README.md)* for the roadmap)
>>>>>> 
>>>>>> I’d appreciate some more reviews & feedback**, but other than
that, I’d be
>>>>>> happy to ship this as an experimental feature in any next release.
>>>>>> 
>>>>>> *
>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>>>>>> **
>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>>>>>> 
>>>>>> 
>>>>>> Best
>>>>>> Jan
>>>>>> --
>>>>>> 
>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>>>> 
>>>>>>> A few updates:
>>>>>>> 
>>>>>>> By Bob Ippolito / @etrepum:
>>>>>>> - Plugins are now installed in libdir (instead of /tmp).
>>>>>>> - Config loading is now done with proper .ini files.
>>>>>>> - Various cleanups and code review (Thanks!).
>>>>>>> 
>>>>>>> Mine (most suggested by Bob):
>>>>>>> - `plugins.html` now shows you if a plugin is already installed.
>>>>>>> and which version, if it doesn’t match the installable one.
>>>>>>> - The Install button now disables after an installation.
>>>>>>> - Plugins are now registered with couch_config as
>>>>>>> /_config/plugins/name = version
>>>>>>> - Updated `couch-config` to print --erlang-version and --erl-bin
>>>>>>> - Updated the geocouch plugin to use the new options in
>>>>>>> `couch-config`.
>>>>>>> - Added Bob Ippolito’s couchperuser plugin to Futon.
>>>>>>> 
>>>>>>> 
>>>>>>> Best
>>>>>>> Jan
>>>>>>> --
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt <jan@apache.org (mailto:jan@apache.org)>
wrote:
>>>>>>> 
>>>>>>>> Heya,
>>>>>>>> 
>>>>>>>> I couldn’t help myself thinking about plugin stuff and
ended up
>>>>>>>> whipping up a proof of concept.
>>>>>>>> 
>>>>>>>> Here’s a <1 minute demo video:
>>>>>>>> 
>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>>>>>>>> 
>>>>>>>> Alternative encoding:
>>>>>>>> 
>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> In my head the whole plugin idea is a very wide area, but
I was so
>>>>>>>> intrigued by the idea of getting something running with a
click on a
>>>>>>>> button in Futon. So I looked for a minimally viable plugin
system.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## Design principles
>>>>>>>> 
>>>>>>>> It took me a day to put this all together and this was only
possible
>>>>>>>> because I took a lot of shortcuts. I believe they are all
viable for a
>>>>>>>> first iteration of a plugins system:
>>>>>>>> 
>>>>>>>> 1. Install with one click on a button in Futon (or HTTP call)
>>>>>>>> 2. Only pure Erlang plugins are allowed.
>>>>>>>> 3. The plugin author must provide a binary package for each
Erlang (and,
>>>>>>>> later, each CouchDB version).
>>>>>>>> 4. Complete trust-based system. You trust me to not do any
nasty things
>>>>>>>> when you click on the install button. No crypto, no nothing.
Only
>>>>>>>> people who can commit to Futon can release new versions of
plugins.
>>>>>>>> 5. Minimal user-friendlyness: won’t install plugins that
don’t match
>>>>>>>> the current Erlang version, gives semi-sensible error messages
>>>>>>>> (wrapped in a HTTP 500 response :)
>>>>>>>> 6. Require a pretty strict format for binary releases.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## Roadmap
>>>>>>>> 
>>>>>>>> Here’s a list of things this first iterations does and
doesn’t do:
>>>>>>>> 
>>>>>>>> - Pure Erlang plugins only. No C-dependencies, no JavaScript,
no
>>>>>> nothing.
>>>>>>>> - No C-dependencies.
>>>>>>>> - Install a plugin via Futon (or HTTP call). Admin only.
>>>>>>>> - A hardcoded list of plugins in Futon.
>>>>>>>> - Loads a pre-packaged, pre-compiled .tar.gz file from a
URL.
>>>>>>>> - Only installs if Erlang version matches.
>>>>>>>> - No security checking of binaries.
>>>>>>>> - No identity checking of binaries.
>>>>>>>> 
>>>>>>>> Here are a few things I want to add before I call it MVP*:
>>>>>>>> 
>>>>>>>> - Uninstall a plugin via Futon (or HTTP call). Admin only.
>>>>>>>> - Only installs if CouchDB version matches.
>>>>>>>> - Binaries must be published on *.apache.org (http://apache.org).
>>>>>>>> - Register installed plugins in the config system.
>>>>>>>> - Make sure plugins start with the next restart of CouchDB.
>>>>>>>> - Show when a particular plugin is installed.
>>>>>>>> 
>>>>>>>> *MVP hopefully means you agree we can ship this with a few
warnings
>>>>>>>> so people can get a hang of it.
>>>>>>>> 
>>>>>>>> Here is a rough list of features squared against future milestones:
>>>>>>>> 
>>>>>>>> Milestone 2: Be creator friendly
>>>>>>>> - Make it easy to build a CouchDB plugin by providing one
or more easy
>>>>>>>> to start templates.
>>>>>>>> - Make it easy to publish new plugins and new versions of
existing
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> plugins.
>>>>>>>> - Make it easy to supply packages for multiple Erlang &
CouchDB
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> versions.
>>>>>>>> 
>>>>>>>> Milestone 3: Public registry
>>>>>>>> - Instead of hardcoding a list of plugins into Futon/Fauxton,
we load
>>>>>>>> a list of applicable plugins from a central (and configurable)
>>>>>>>> plugins repository.
>>>>>>>> - This allows plugin authors to publish new plugins and new
versions
>>>>>>>> of existing plugins independently.
>>>>>>>> 
>>>>>>>> Milestone 4: Other Languages
>>>>>>>> - Figure out how to handle C-dependencies for Erlang plugins.
>>>>>>>> - Figure out how to allow other language plugins
>>>>>>>> (c.f. non-JS query servers)
>>>>>>>> 
>>>>>>>> Milestone X: Later
>>>>>>>> - Add some account/identity/maybe crypto-web-of-trust system
for
>>>>>>>> authors to publish “legit” plugins.
>>>>>>>> - Sign & verify individual releases.
>>>>>>>> 
>>>>>>>> A few more things that can happen concurrently depending
on what
>>>>>>>> plugins require:
>>>>>>>> - Integrate Erlang/JS tests in the installation
>>>>>>>> - Integrate docs
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## How it works
>>>>>>>> 
>>>>>>>> This plugin system lives in `src/couch_plugins` and is a
tiny CouchDB
>>>>>>>> module.
>>>>>>>> 
>>>>>>>> It exposes one new API endpoint `/_plugins` that an admin
user can
>>>>>>>> POST to.
>>>>>>>> 
>>>>>>>> The additional Futon page lives at /_utils/plugins.html it
is
>>>>>>>> hardcoded.
>>>>>>>> 
>>>>>>>> Futon (or you) post an object to `/_plugins` with four properties:
>>>>>>>> 
>>>>>>>> {
>>>>>>>> "name": "geocouch", // name of the plugin, must be unique
>>>>>>>> "url": "http://people.apache.org/~jan", // “base URL”
for plugin
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> releases (see below)
>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22", // whatever
version
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> internal to the plugin
>>>>>>>> "checksums": {
>>>>>>>> "R15B03": "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d sha
hash
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> over the binary
>>>>>>>> }
>>>>>>>> }
>>>>>>>> 
>>>>>>>> `couch_plugins` then attempts to download a .tar.gz from
this
>>>>>>>> location:
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>>>>>>>> 
>>>>>>>> It should be obvious how the URL is constructed from the
POST data.
>>>>>>>> (This url is live, feel free to play around with this tarball).
>>>>>>>> 
>>>>>>>> Next it calculates the sha hash for the downloaded .tar.gz
file and
>>>>>>>> matches it against the correct version in the `checksums`
parameter.
>>>>>>>> 
>>>>>>>> If that succeeds, we unpack the .tar.gz file (currently in
`/tmp`,
>>>>>>>> need to find a better place for this) and adds the extracted
directory
>>>>>>>> to the Erlang code path
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>>>>>>>> and loads the included application (`application:load(geocouch)`).
>>>>>>>> 
>>>>>>>> Then it looks into the `./config` directory that lives next
to `ebin/`
>>>>>>>> in the plugin directory for a file `config.erlt` (“erl-terms”).
with a
>>>>>>>> list of configuration parameters to load. We parse the file
and set
>>>>>>>> the config directives one by one.
>>>>>>>> 
>>>>>>>> If that all goes to plan, we report success back to the HTTP
caller.
>>>>>>>> 
>>>>>>>> That’s it! :)
>>>>>>>> 
>>>>>>>> It’s deceptively simple, probably does a few things very
wrong and
>>>>>>>> leaves a few things open (see above).
>>>>>>>> 
>>>>>>>> One open question I’d like an answer for is finding a good
location to
>>>>>>>> unpack & install the plugin files that isn’t `tmp`.
If the answer is
>>>>>>>> different for a pre-BigCouch/rcouch-merge and post-BigCouch/rcouch-
>>>>>>>> merge world, I’d love to know :)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## Code
>>>>>>>> 
>>>>>>>> The main branch for this is 1867-feature-plugins:
>>>>>>>> 
>>>>>>>> ASF:
>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>>>>>>>> GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>>>>>>>> 
>>>>>>>> I created a branch on GeoCouch that adds a few lines to its
`Makefile`
>>>>>>>> that shows how a binary package is built:
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>>>>>>>> 
>>>>>>>> * * *
>>>>>>>> 
>>>>>>>> I hope you like this :) Please comment and improve heavily!
>>>>>>>> 
>>>>>>>> Let me know if you have any questions :)
>>>>>>>> 
>>>>>>>> If you have any criticism, please phrase it in a way that
we can use
>>>>>>>> to improve this, thanks!
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Jan
>>>>>>>> --
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message