couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Octavian Damiean <mainer...@gmail.com>
Subject Re: Minimally Viable Plugins
Date Sun, 04 Aug 2013 16:03:47 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Oh yea, forgot to tell. Done. :)

On 2013-08-04 17:39, Jan Lehnardt wrote:
> 
> On Aug 4, 2013, at 17:29 , Octavian Damiean <mainerror@gmail.com>
> wrote:
> 
>> Signed PGP part Great work!
>> 
>> Got a quick question. In the notice you mention that the
>> instructions work only with the 1867-feature-plugin branch but
>> you didn't add steps to checkout said branch in the preparation
>> step.
>> 
>> Want me to add that bit of information or was that deliberate?
> 
> Total oversight, you are more then welcome to add this :)
> 
> Jan --
> 
> 
> 
>> 
>> Cheers, Octavian
>> 
>> On 2013-08-03 22:15, Jan Lehnardt wrote:
>>> More update:
>>> 
>>> I started producing a plugin template repo that people can
>>> clone to build their own plugins along with a comprehensive
>>> README.md:
>>> 
>>> https://github.com/janl/my-first-couchdb-plugin
>>> 
>>> The idea is to move the README to the CouchDB docs eventually
>>> and ship the plugin template with CouchDB, so people can get
>>> started easily.
>>> 
>>> Best Jan --
>>> 
>>> On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com> 
>>> wrote:
>>> 
>>>> :)
>>>> 
>>>> 
>>>> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
>>>> 
>>>>> Couldn’t help but implement it. It’s in the branch now.
>>>>> 
>>>>> Jan --
>>>>> 
>>>>> On Aug 3, 2013, at 08:12 , Simon Metson
>>>>> <simon@cloudant.com> wrote:
>>>>> 
>>>>>> Sounds good to me.
>>>>>> 
>>>>>> 
>>>>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Aug 3, 2013, at 00:02 , Russell Branca 
>>>>>>> <chewbranca@apache.org (mailto:chewbranca@apache.org)>

>>>>>>> wrote:
>>>>>>> 
>>>>>>>> This is fantastic, Jan! Glad to see this coming
>>>>>>>> along.
>>>>>>>> 
>>>>>>>> One of the goals with Fauxton has always been to make
>>>>>>>> it easy for plugins to extend the interface and
>>>>>>>> provide new functionality. I've been toying with the
>>>>>>>> idea of having a _fauxton db that plugins install to
>>>>>>>> as docs with attachments, but that's for a different
>>>>>>>> thread. The general idea here is that a plugin will
>>>>>>>> be able to extend Fauxton by adding a new page with
>>>>>>>> it's own functionality, or hook into existing pages
>>>>>>>> to extend other areas.
>>>>>>>> 
>>>>>>>> For instance, you could have a couchdb-lucene plugin
>>>>>>>> that hooks into the databases list and allows you to
>>>>>>>> add interfaces for building full text indexes and
>>>>>>>> searching on existing indexes. Or you could have a
>>>>>>>> dedicated page for Geocouch, or whatever.
>>>>>>>> 
>>>>>>>> The functionality is there, but it's still a bit of
>>>>>>>> a manual process, so we'll need to make it more
>>>>>>>> dynamic and smooth out the rough edges.
>>>>>>>> 
>>>>>>>> I'm very excited to see progress being made on
>>>>>>>> plugins, great work!
>>>>>>> 
>>>>>>> Thanks, I’m glad you like this! :)
>>>>>>> 
>>>>>>> Another way to get the Fauxton plugin loaded would be
>>>>>>> to extend the /_plugins API endpoint, so Fauxton could
>>>>>>> request GET /_plugins/<pluginname>/ and it would serve

>>>>>>> <couchdblibdir>/plugins/<pluginname/priv/www which is
>>>>>>> just a place for Fauxton-enabled plugins.
>>>>>>> 
>>>>>>> Fauxton would walk /_config/plugins/ to get to a list
>>>>>>> of plugins.
>>>>>>> 
>>>>>>> In fact that should be pretty simple to set up.
>>>>>>> 
>>>>>>> For now I am trying to avoid having a custom database
>>>>>>> for this, mostly because I don’t think there are many 
>>>>>>> advantages (e.g. replication of plugins?) and code 
>>>>>>> complexity. These priorities might change in the
>>>>>>> future, but for now I am happy to get this working at
>>>>>>> all :)
>>>>>>> 
>>>>>>> If you are okay with the above plan of serving plugin 
>>>>>>> HTML/JS/CSS from /_plugins/<pluginname>, I’m happy to
>>>>>>> add this to the branch.
>>>>>>> 
>>>>>>> Best Jan --
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -Russell
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt 
>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>> 
>>>>>>>>> And a few more (from COUCHDB-1867):
>>>>>>>>> 
>>>>>>>>> - Add uninstall, incl. Futon UI. - Only install a 
>>>>>>>>> plugin if the source and target CouchDB version 
>>>>>>>>> matches. - Rebase against master.
>>>>>>>>> 
>>>>>>>>> * * *
>>>>>>>>> 
>>>>>>>>> This concludes my list for a Minimally Viable
>>>>>>>>> Plugin feature. (See the original email or
>>>>>>>>> README.md (http://README.md)* for the roadmap)
>>>>>>>>> 
>>>>>>>>> I’d appreciate some more reviews & feedback**,
but 
>>>>>>>>> other than that, I’d be happy to ship this as an 
>>>>>>>>> experimental feature in any next release.
>>>>>>>>> 
>>>>>>>>> * 
>>>>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
**
>>>>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
Best
>>>>>>>>> Jan --
>>>>>>>>> 
>>>>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt
>>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>>> 
>>>>>>>>>> A few updates:
>>>>>>>>>> 
>>>>>>>>>> By Bob Ippolito / @etrepum: - Plugins are now 
>>>>>>>>>> installed in libdir (instead of /tmp). - Config 
>>>>>>>>>> loading is now done with proper .ini files. -
>>>>>>>>>> Various cleanups and code review (Thanks!).
>>>>>>>>>> 
>>>>>>>>>> Mine (most suggested by Bob): - `plugins.html`
>>>>>>>>>> now shows you if a plugin is already installed.
>>>>>>>>>> and which version, if it doesn’t match the
>>>>>>>>>> installable one. - The Install button now
>>>>>>>>>> disables after an installation. - Plugins are now
>>>>>>>>>> registered with couch_config as
>>>>>>>>>> /_config/plugins/name = version - Updated
>>>>>>>>>> `couch-config` to print --erlang-version and 
>>>>>>>>>> --erl-bin - Updated the geocouch plugin to use
>>>>>>>>>> the new options in `couch-config`. - Added Bob
>>>>>>>>>> Ippolito’s couchperuser plugin to Futon.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Best Jan --
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt 
>>>>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Heya,
>>>>>>>>>>> 
>>>>>>>>>>> I couldn’t help myself thinking about plugin
>>>>>>>>>>> stuff and ended up whipping up a proof of
>>>>>>>>>>> concept.
>>>>>>>>>>> 
>>>>>>>>>>> Here’s a <1 minute demo video:
>>>>>>>>>>> 
>>>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>
>>>>>>>>>>> 
Alternative encoding:
>>>>>>>>>>> 
>>>>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>
>>>>>>>>>>> 
In my head the whole plugin idea is a very wide area, but I was so
>>>>>>>>>>> intrigued by the idea of getting something
>>>>>>>>>>> running with a click on a button in Futon. So
I
>>>>>>>>>>> looked for a minimally viable plugin system.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ## Design principles
>>>>>>>>>>> 
>>>>>>>>>>> It took me a day to put this all together and
>>>>>>>>>>> this was only possible because I took a lot of

>>>>>>>>>>> shortcuts. I believe they are all viable for
a
>>>>>>>>>>>  first iteration of a plugins system:
>>>>>>>>>>> 
>>>>>>>>>>> 1. Install with one click on a button in Futon
>>>>>>>>>>> (or HTTP call) 2. Only pure Erlang plugins are

>>>>>>>>>>> allowed. 3. The plugin author must provide a
>>>>>>>>>>> binary package for each Erlang (and, later,
>>>>>>>>>>> each CouchDB version). 4. Complete trust-based
>>>>>>>>>>> system. You trust me to not do any nasty things
>>>>>>>>>>> when you click on the install button. No
>>>>>>>>>>> crypto, no nothing. Only people who can commit
>>>>>>>>>>> to Futon can release new versions of plugins.
>>>>>>>>>>> 5. Minimal user-friendlyness: won’t install
>>>>>>>>>>> plugins that don’t match the current Erlang

>>>>>>>>>>> version, gives semi-sensible error messages 
>>>>>>>>>>> (wrapped in a HTTP 500 response :) 6. Require
>>>>>>>>>>> a pretty strict format for binary releases.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ## Roadmap
>>>>>>>>>>> 
>>>>>>>>>>> Here’s a list of things this first iterations
>>>>>>>>>>> does and doesn’t do:
>>>>>>>>>>> 
>>>>>>>>>>> - Pure Erlang plugins only. No C-dependencies,
>>>>>>>>>>> no JavaScript, no
>>>>>>>>> nothing.
>>>>>>>>>>> - No C-dependencies. - Install a plugin via
>>>>>>>>>>> Futon (or HTTP call). Admin only. - A hardcoded
>>>>>>>>>>> list of plugins in Futon. - Loads a
>>>>>>>>>>> pre-packaged, pre-compiled .tar.gz file from
a
>>>>>>>>>>> URL. - Only installs if Erlang version matches.
>>>>>>>>>>> - No security checking of binaries. - No
>>>>>>>>>>> identity checking of binaries.
>>>>>>>>>>> 
>>>>>>>>>>> Here are a few things I want to add before I
>>>>>>>>>>> call it MVP*:
>>>>>>>>>>> 
>>>>>>>>>>> - Uninstall a plugin via Futon (or HTTP call).

>>>>>>>>>>> Admin only. - Only installs if CouchDB version

>>>>>>>>>>> matches. - Binaries must be published on 
>>>>>>>>>>> *.apache.org (http://apache.org). - Register

>>>>>>>>>>> installed plugins in the config system. - Make
>>>>>>>>>>> sure plugins start with the next restart of
>>>>>>>>>>> CouchDB. - Show when a particular plugin is
>>>>>>>>>>> installed.
>>>>>>>>>>> 
>>>>>>>>>>> *MVP hopefully means you agree we can ship
>>>>>>>>>>> this with a few warnings so people can get a
>>>>>>>>>>> hang of it.
>>>>>>>>>>> 
>>>>>>>>>>> Here is a rough list of features squared
>>>>>>>>>>> against future milestones:
>>>>>>>>>>> 
>>>>>>>>>>> Milestone 2: Be creator friendly - Make it easy
>>>>>>>>>>> to build a CouchDB plugin by providing one or
>>>>>>>>>>> more easy to start templates. - Make it easy
to
>>>>>>>>>>> publish new plugins and new versions of
>>>>>>>>>>> existing
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> plugins.
>>>>>>>>>>> - Make it easy to supply packages for multiple

>>>>>>>>>>> Erlang & CouchDB
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> versions.
>>>>>>>>>>> 
>>>>>>>>>>> Milestone 3: Public registry - Instead of 
>>>>>>>>>>> hardcoding a list of plugins into
>>>>>>>>>>> Futon/Fauxton, we load a list of applicable
>>>>>>>>>>> plugins from a central (and configurable)
>>>>>>>>>>> plugins repository. - This allows plugin
>>>>>>>>>>> authors to publish new plugins and new versions
>>>>>>>>>>> of existing plugins independently.
>>>>>>>>>>> 
>>>>>>>>>>> Milestone 4: Other Languages - Figure out how
>>>>>>>>>>> to handle C-dependencies for Erlang plugins.
-
>>>>>>>>>>> Figure out how to allow other language plugins
>>>>>>>>>>> (c.f. non-JS query servers)
>>>>>>>>>>> 
>>>>>>>>>>> Milestone X: Later - Add some 
>>>>>>>>>>> account/identity/maybe crypto-web-of-trust
>>>>>>>>>>> system for authors to publish “legit” plugins.
>>>>>>>>>>> - Sign & verify individual releases.
>>>>>>>>>>> 
>>>>>>>>>>> A few more things that can happen concurrently

>>>>>>>>>>> depending on what plugins require: - Integrate

>>>>>>>>>>> Erlang/JS tests in the installation -
>>>>>>>>>>> Integrate docs
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ## How it works
>>>>>>>>>>> 
>>>>>>>>>>> This plugin system lives in `src/couch_plugins`
>>>>>>>>>>> and is a tiny CouchDB module.
>>>>>>>>>>> 
>>>>>>>>>>> It exposes one new API endpoint `/_plugins`
>>>>>>>>>>> that an admin user can POST to.
>>>>>>>>>>> 
>>>>>>>>>>> The additional Futon page lives at 
>>>>>>>>>>> /_utils/plugins.html it is hardcoded.
>>>>>>>>>>> 
>>>>>>>>>>> Futon (or you) post an object to `/_plugins`
>>>>>>>>>>> with four properties:
>>>>>>>>>>> 
>>>>>>>>>>> { "name": "geocouch", // name of the plugin,
>>>>>>>>>>> must be unique "url":
>>>>>>>>>>> "http://people.apache.org/~jan", // “base URL”
>>>>>>>>>>> for plugin
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> releases (see below)
>>>>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22",
>>>>>>>>>>> // whatever version
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> internal to the plugin
>>>>>>>>>>> "checksums": { "R15B03": 
>>>>>>>>>>> "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d
sha 
>>>>>>>>>>> hash
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> over the binary
>>>>>>>>>>> } }
>>>>>>>>>>> 
>>>>>>>>>>> `couch_plugins` then attempts to download a
>>>>>>>>>>> .tar.gz from this location:
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
It should be obvious how the URL is constructed from the POST data.
>>>>>>>>>>> (This url is live, feel free to play around
>>>>>>>>>>> with this tarball).
>>>>>>>>>>> 
>>>>>>>>>>> Next it calculates the sha hash for the
>>>>>>>>>>> downloaded .tar.gz file and matches it against
>>>>>>>>>>> the correct version in the `checksums`
>>>>>>>>>>> parameter.
>>>>>>>>>>> 
>>>>>>>>>>> If that succeeds, we unpack the .tar.gz file

>>>>>>>>>>> (currently in `/tmp`, need to find a better
>>>>>>>>>>> place for this) and adds the extracted
>>>>>>>>>>> directory to the Erlang code path
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>>>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
and loads the included application (`application:load(geocouch)`).
>>>>>>>>>>> 
>>>>>>>>>>> Then it looks into the `./config` directory
>>>>>>>>>>> that lives next to `ebin/` in the plugin
>>>>>>>>>>> directory for a file `config.erlt`
>>>>>>>>>>> (“erl-terms”). with a list of configuration
>>>>>>>>>>> parameters to load. We parse the file and set
>>>>>>>>>>> the config directives one by one.
>>>>>>>>>>> 
>>>>>>>>>>> If that all goes to plan, we report success
>>>>>>>>>>> back to the HTTP caller.
>>>>>>>>>>> 
>>>>>>>>>>> That’s it! :)
>>>>>>>>>>> 
>>>>>>>>>>> It’s deceptively simple, probably does a few
>>>>>>>>>>> things very wrong and leaves a few things open
>>>>>>>>>>> (see above).
>>>>>>>>>>> 
>>>>>>>>>>> One open question I’d like an answer for is
>>>>>>>>>>> finding a good location to unpack & install
the
>>>>>>>>>>> plugin files that isn’t `tmp`. If the answer
is
>>>>>>>>>>> different for a pre-BigCouch/rcouch-merge and

>>>>>>>>>>> post-BigCouch/rcouch- merge world, I’d love
to
>>>>>>>>>>> know :)
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> ## Code
>>>>>>>>>>> 
>>>>>>>>>>> The main branch for this is
>>>>>>>>>>> 1867-feature-plugins:
>>>>>>>>>>> 
>>>>>>>>>>> ASF:
>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>>>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>>>>>>>>>>> 
>>>>>>>>>>> I created a branch on GeoCouch that adds a few

>>>>>>>>>>> lines to its `Makefile` that shows how a
>>>>>>>>>>> binary package is built:
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>
>>>>>>>>> 
* * *
>>>>>>>>>>> 
>>>>>>>>>>> I hope you like this :) Please comment and
>>>>>>>>>>> improve heavily!
>>>>>>>>>>> 
>>>>>>>>>>> Let me know if you have any questions :)
>>>>>>>>>>> 
>>>>>>>>>>> If you have any criticism, please phrase it in
>>>>>>>>>>> a way that we can use to improve this, thanks!
>>>>>>>>>>> 
>>>>>>>>>>> Best, Jan --
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJR/ntiAAoJEAq942X94y8mpkYIAK1w9MBOmXSw3BZc9OJf55W1
4KRHvtvpyUy5BTZyoCHCeKquHDIy+rRJTnX69y3k4yXwEAG6kocJ5UldWUSn/tX8
RAnEyIeTQs5YAjybutPIjpw6ClXgJnxON4sDCjvDPMoj3csE2RbxHqMD1udSGDd5
OuUdqYBvC2g8XpNq3hbxtX7O0982s4OOfdJAywTpQ4vrcBhql+5m70YUiM+zpRF2
JjQxngk9110aMEBDm6CgjW6z7XtwMbWqsXuFDfr5/RvFwBdAD+PksfWvPyUOmq2x
Bga7Dqa+clBN6AmA6oNGibXcIAjL//vqynFK4XKOSBchKuLurPypE5b30pkzmX4=
=S/C3
-----END PGP SIGNATURE-----

Mime
View raw message