incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Wenk <a...@nms.de>
Subject Re: Minimally Viable Plugins
Date Sun, 04 Aug 2013 15:50:17 GMT
wow - awesome - just wanted to let you know that this will be really
cool - having plugins will increase the CouchDB users a lot! Great
job!

On 4 August 2013 17:39, Jan Lehnardt <jan@apache.org> wrote:
>
> On Aug 4, 2013, at 17:29 , Octavian Damiean <mainerror@gmail.com> wrote:
>
>> Signed PGP part
>> Great work!
>>
>> Got a quick question. In the notice you mention that the instructions
>> work only with the 1867-feature-plugin branch but you didn't add steps
>> to checkout said branch in the preparation step.
>>
>> Want me to add that bit of information or was that deliberate?
>
> Total oversight, you are more then welcome to add this :)
>
> Jan
> --
>
>
>
>>
>> Cheers,
>> Octavian
>>
>> On 2013-08-03 22:15, Jan Lehnardt wrote:
>> > More update:
>> >
>> > I started producing a plugin template repo that people can clone
>> > to build their own plugins along with a comprehensive README.md:
>> >
>> > https://github.com/janl/my-first-couchdb-plugin
>> >
>> > The idea is to move the README to the CouchDB docs eventually and
>> > ship the plugin template with CouchDB, so people can get started
>> > easily.
>> >
>> > Best Jan --
>> >
>> > On Aug 3, 2013, at 17:53 , Simon Metson <simon@cloudant.com>
>> > wrote:
>> >
>> >> :)
>> >>
>> >>
>> >> On Saturday, 3 August 2013 at 14:21, Jan Lehnardt wrote:
>> >>
>> >>> Couldn’t help but implement it. It’s in the branch now.
>> >>>
>> >>> Jan --
>> >>>
>> >>> On Aug 3, 2013, at 08:12 , Simon Metson <simon@cloudant.com>
>> >>> wrote:
>> >>>
>> >>>> Sounds good to me.
>> >>>>
>> >>>>
>> >>>> On Saturday, 3 August 2013 at 00:56, Jan Lehnardt wrote:
>> >>>>
>> >>>>>
>> >>>>> On Aug 3, 2013, at 00:02 , Russell Branca
>> >>>>> <chewbranca@apache.org (mailto:chewbranca@apache.org)>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> This is fantastic, Jan! Glad to see this coming along.
>> >>>>>>
>> >>>>>> One of the goals with Fauxton has always been to make it
>> >>>>>> easy for plugins to extend the interface and provide new
>> >>>>>> functionality. I've been toying with the idea of having
a
>> >>>>>> _fauxton db that plugins install to as docs with
>> >>>>>> attachments, but that's for a different thread. The
>> >>>>>> general idea here is that a plugin will be able to extend
>> >>>>>> Fauxton by adding a new page with it's own functionality,
>> >>>>>> or hook into existing pages to extend other areas.
>> >>>>>>
>> >>>>>> For instance, you could have a couchdb-lucene plugin that
>> >>>>>> hooks into the databases list and allows you to add
>> >>>>>> interfaces for building full text indexes and searching
>> >>>>>> on existing indexes. Or you could have a dedicated page
>> >>>>>> for Geocouch, or whatever.
>> >>>>>>
>> >>>>>> The functionality is there, but it's still a bit of a
>> >>>>>> manual process, so we'll need to make it more dynamic and
>> >>>>>> smooth out the rough edges.
>> >>>>>>
>> >>>>>> I'm very excited to see progress being made on plugins,
>> >>>>>> great work!
>> >>>>>
>> >>>>> Thanks, I’m glad you like this! :)
>> >>>>>
>> >>>>> Another way to get the Fauxton plugin loaded would be to
>> >>>>> extend the /_plugins API endpoint, so Fauxton could request
>> >>>>> GET /_plugins/<pluginname>/ and it would serve
>> >>>>> <couchdblibdir>/plugins/<pluginname/priv/www which
is just
>> >>>>> a place for Fauxton-enabled plugins.
>> >>>>>
>> >>>>> Fauxton would walk /_config/plugins/ to get to a list of
>> >>>>> plugins.
>> >>>>>
>> >>>>> In fact that should be pretty simple to set up.
>> >>>>>
>> >>>>> For now I am trying to avoid having a custom database for
>> >>>>> this, mostly because I don’t think there are many
>> >>>>> advantages (e.g. replication of plugins?) and code
>> >>>>> complexity. These priorities might change in the future,
>> >>>>> but for now I am happy to get this working at all :)
>> >>>>>
>> >>>>> If you are okay with the above plan of serving plugin
>> >>>>> HTML/JS/CSS from /_plugins/<pluginname>, I’m happy to
add
>> >>>>> this to the branch.
>> >>>>>
>> >>>>> Best Jan --
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> -Russell
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Aug 2, 2013 at 2:17 PM, Jan Lehnardt
>> >>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>> >>>>>>
>> >>>>>>> And a few more (from COUCHDB-1867):
>> >>>>>>>
>> >>>>>>> - Add uninstall, incl. Futon UI. - Only install a
>> >>>>>>> plugin if the source and target CouchDB version
>> >>>>>>> matches. - Rebase against master.
>> >>>>>>>
>> >>>>>>> * * *
>> >>>>>>>
>> >>>>>>> This concludes my list for a Minimally Viable Plugin
>> >>>>>>> feature. (See the original email or README.md
>> >>>>>>> (http://README.md)* for the roadmap)
>> >>>>>>>
>> >>>>>>> I’d appreciate some more reviews & feedback**,
but
>> >>>>>>> other than that, I’d be happy to ship this as an
>> >>>>>>> experimental feature in any next release.
>> >>>>>>>
>> >>>>>>> *
>> >>>>>>> https://github.com/janl/couchdb/blob/1867-feature-plugins/src/couch_plugins/README.md#roadmap
>> >>>>>>>
>> >>>>>>>
>> **
>> >>>>>>> https://github.com/janl/couchdb/compare/apache:master...1867-feature-plugins
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> Best
>> >>>>>>> Jan --
>> >>>>>>>
>> >>>>>>> On Aug 1, 2013, at 19:34 , Jan Lehnardt <jan@apache.org
>> >>>>>>> (mailto:jan@apache.org)> wrote:
>> >>>>>>>
>> >>>>>>>> A few updates:
>> >>>>>>>>
>> >>>>>>>> By Bob Ippolito / @etrepum: - Plugins are now
>> >>>>>>>> installed in libdir (instead of /tmp). - Config
>> >>>>>>>> loading is now done with proper .ini files. - Various
>> >>>>>>>> cleanups and code review (Thanks!).
>> >>>>>>>>
>> >>>>>>>> Mine (most suggested by Bob): - `plugins.html` now
>> >>>>>>>> shows you if a plugin is already installed. and
which
>> >>>>>>>> version, if it doesn’t match the installable one.
-
>> >>>>>>>> The Install button now disables after an
>> >>>>>>>> installation. - Plugins are now registered with
>> >>>>>>>> couch_config as /_config/plugins/name = version
-
>> >>>>>>>> Updated `couch-config` to print --erlang-version
and
>> >>>>>>>> --erl-bin - Updated the geocouch plugin to use the
>> >>>>>>>> new options in `couch-config`. - Added Bob Ippolito’s
>> >>>>>>>> couchperuser plugin to Futon.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Best Jan --
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Jul 31, 2013, at 19:07 , Jan Lehnardt
>> >>>>>>>> <jan@apache.org (mailto:jan@apache.org)> wrote:
>> >>>>>>>>
>> >>>>>>>>> Heya,
>> >>>>>>>>>
>> >>>>>>>>> I couldn’t help myself thinking about plugin
stuff
>> >>>>>>>>> and ended up whipping up a proof of concept.
>> >>>>>>>>>
>> >>>>>>>>> Here’s a <1 minute demo video:
>> >>>>>>>>>
>> >>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.mov
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> Alternative encoding:
>> >>>>>>>>>
>> >>>>>>>>> https://dl.dropboxusercontent.com/u/82149/couchdb-plugins-demo.m4v)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> In my head the whole plugin idea is a very wide area, but I was so
>> >>>>>>>>> intrigued by the idea of getting something running
>> >>>>>>>>> with a click on a button in Futon. So I looked
for
>> >>>>>>>>> a minimally viable plugin system.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ## Design principles
>> >>>>>>>>>
>> >>>>>>>>> It took me a day to put this all together and
this
>> >>>>>>>>> was only possible because I took a lot of
>> >>>>>>>>> shortcuts. I believe they are all viable for
a
>> >>>>>>>>> first iteration of a plugins system:
>> >>>>>>>>>
>> >>>>>>>>> 1. Install with one click on a button in Futon
(or
>> >>>>>>>>> HTTP call) 2. Only pure Erlang plugins are
>> >>>>>>>>> allowed. 3. The plugin author must provide a
binary
>> >>>>>>>>> package for each Erlang (and, later, each CouchDB
>> >>>>>>>>> version). 4. Complete trust-based system. You
trust
>> >>>>>>>>> me to not do any nasty things when you click
on the
>> >>>>>>>>> install button. No crypto, no nothing. Only
people
>> >>>>>>>>> who can commit to Futon can release new versions
of
>> >>>>>>>>> plugins. 5. Minimal user-friendlyness: won’t
>> >>>>>>>>> install plugins that don’t match the current
Erlang
>> >>>>>>>>> version, gives semi-sensible error messages
>> >>>>>>>>> (wrapped in a HTTP 500 response :) 6. Require
a
>> >>>>>>>>> pretty strict format for binary releases.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ## Roadmap
>> >>>>>>>>>
>> >>>>>>>>> Here’s a list of things this first iterations
does
>> >>>>>>>>> and doesn’t do:
>> >>>>>>>>>
>> >>>>>>>>> - Pure Erlang plugins only. No C-dependencies,
no
>> >>>>>>>>> JavaScript, no
>> >>>>>>> nothing.
>> >>>>>>>>> - No C-dependencies. - Install a plugin via
Futon
>> >>>>>>>>> (or HTTP call). Admin only. - A hardcoded list
of
>> >>>>>>>>> plugins in Futon. - Loads a pre-packaged,
>> >>>>>>>>> pre-compiled .tar.gz file from a URL. - Only
>> >>>>>>>>> installs if Erlang version matches. - No security
>> >>>>>>>>> checking of binaries. - No identity checking
of
>> >>>>>>>>> binaries.
>> >>>>>>>>>
>> >>>>>>>>> Here are a few things I want to add before I
call
>> >>>>>>>>> it MVP*:
>> >>>>>>>>>
>> >>>>>>>>> - Uninstall a plugin via Futon (or HTTP call).
>> >>>>>>>>> Admin only. - Only installs if CouchDB version
>> >>>>>>>>> matches. - Binaries must be published on
>> >>>>>>>>> *.apache.org (http://apache.org). - Register
>> >>>>>>>>> installed plugins in the config system. - Make
sure
>> >>>>>>>>> plugins start with the next restart of CouchDB.
-
>> >>>>>>>>> Show when a particular plugin is installed.
>> >>>>>>>>>
>> >>>>>>>>> *MVP hopefully means you agree we can ship this
>> >>>>>>>>> with a few warnings so people can get a hang
of
>> >>>>>>>>> it.
>> >>>>>>>>>
>> >>>>>>>>> Here is a rough list of features squared against
>> >>>>>>>>> future milestones:
>> >>>>>>>>>
>> >>>>>>>>> Milestone 2: Be creator friendly - Make it easy
to
>> >>>>>>>>> build a CouchDB plugin by providing one or more
>> >>>>>>>>> easy to start templates. - Make it easy to publish
>> >>>>>>>>> new plugins and new versions of existing
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> plugins.
>> >>>>>>>>> - Make it easy to supply packages for multiple
>> >>>>>>>>> Erlang & CouchDB
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> versions.
>> >>>>>>>>>
>> >>>>>>>>> Milestone 3: Public registry - Instead of
>> >>>>>>>>> hardcoding a list of plugins into Futon/Fauxton,
we
>> >>>>>>>>> load a list of applicable plugins from a central
>> >>>>>>>>> (and configurable) plugins repository. - This
>> >>>>>>>>> allows plugin authors to publish new plugins
and
>> >>>>>>>>> new versions of existing plugins independently.
>> >>>>>>>>>
>> >>>>>>>>> Milestone 4: Other Languages - Figure out how
to
>> >>>>>>>>> handle C-dependencies for Erlang plugins. -
Figure
>> >>>>>>>>> out how to allow other language plugins (c.f.
>> >>>>>>>>> non-JS query servers)
>> >>>>>>>>>
>> >>>>>>>>> Milestone X: Later - Add some
>> >>>>>>>>> account/identity/maybe crypto-web-of-trust system
>> >>>>>>>>> for authors to publish “legit” plugins.
- Sign &
>> >>>>>>>>> verify individual releases.
>> >>>>>>>>>
>> >>>>>>>>> A few more things that can happen concurrently
>> >>>>>>>>> depending on what plugins require: - Integrate
>> >>>>>>>>> Erlang/JS tests in the installation - Integrate
>> >>>>>>>>> docs
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ## How it works
>> >>>>>>>>>
>> >>>>>>>>> This plugin system lives in `src/couch_plugins`
and
>> >>>>>>>>> is a tiny CouchDB module.
>> >>>>>>>>>
>> >>>>>>>>> It exposes one new API endpoint `/_plugins`
that an
>> >>>>>>>>> admin user can POST to.
>> >>>>>>>>>
>> >>>>>>>>> The additional Futon page lives at
>> >>>>>>>>> /_utils/plugins.html it is hardcoded.
>> >>>>>>>>>
>> >>>>>>>>> Futon (or you) post an object to `/_plugins`
with
>> >>>>>>>>> four properties:
>> >>>>>>>>>
>> >>>>>>>>> { "name": "geocouch", // name of the plugin,
must
>> >>>>>>>>> be unique "url": "http://people.apache.org/~jan",
>> >>>>>>>>> // “base URL” for plugin
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> releases (see below)
>> >>>>>>>>> "version": "couchdb1.2.x_v0.3.0-11-gd83ba22",
//
>> >>>>>>>>> whatever version
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> internal to the plugin
>> >>>>>>>>> "checksums": { "R15B03":
>> >>>>>>>>> "ZetgdHj2bY2w37buulWVf3USOZs=" // base64’d
sha
>> >>>>>>>>> hash
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> over the binary
>> >>>>>>>>> } }
>> >>>>>>>>>
>> >>>>>>>>> `couch_plugins` then attempts to download a
.tar.gz
>> >>>>>>>>> from this location:
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> http://people.apache.org/~jan/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03.tar.gz
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> It should be obvious how the URL is constructed from the POST data.
>> >>>>>>>>> (This url is live, feel free to play around
with
>> >>>>>>>>> this tarball).
>> >>>>>>>>>
>> >>>>>>>>> Next it calculates the sha hash for the downloaded
>> >>>>>>>>> .tar.gz file and matches it against the correct
>> >>>>>>>>> version in the `checksums` parameter.
>> >>>>>>>>>
>> >>>>>>>>> If that succeeds, we unpack the .tar.gz file
>> >>>>>>>>> (currently in `/tmp`, need to find a better
place
>> >>>>>>>>> for this) and adds the extracted directory to
the
>> >>>>>>>>> Erlang code path
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> (`code:add_path("/tmp/couchdb_plugins/geocouch-couchdb1.2.x_v0.3.0-12-g4ea0bea-R15B03/ebin")`)
>> >>>>>>>>>
>> >>>>>>>
>> and loads the included application (`application:load(geocouch)`).
>> >>>>>>>>>
>> >>>>>>>>> Then it looks into the `./config` directory
that
>> >>>>>>>>> lives next to `ebin/` in the plugin directory
for a
>> >>>>>>>>> file `config.erlt` (“erl-terms”). with a
list of
>> >>>>>>>>> configuration parameters to load. We parse the
file
>> >>>>>>>>> and set the config directives one by one.
>> >>>>>>>>>
>> >>>>>>>>> If that all goes to plan, we report success
back to
>> >>>>>>>>> the HTTP caller.
>> >>>>>>>>>
>> >>>>>>>>> That’s it! :)
>> >>>>>>>>>
>> >>>>>>>>> It’s deceptively simple, probably does a few
things
>> >>>>>>>>> very wrong and leaves a few things open (see
>> >>>>>>>>> above).
>> >>>>>>>>>
>> >>>>>>>>> One open question I’d like an answer for is
finding
>> >>>>>>>>> a good location to unpack & install the
plugin
>> >>>>>>>>> files that isn’t `tmp`. If the answer is different
>> >>>>>>>>> for a pre-BigCouch/rcouch-merge and
>> >>>>>>>>> post-BigCouch/rcouch- merge world, I’d love
to know
>> >>>>>>>>> :)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ## Code
>> >>>>>>>>>
>> >>>>>>>>> The main branch for this is 1867-feature-plugins:
>> >>>>>>>>>
>> >>>>>>>>> ASF:
>> >>>>>>> https://git-wip-us.apache.org/repos/asf?p=couchdb.git;a=log;h=refs/heads/1867-feature-plugins
>> >>>>>>>>>
>> >>>>>>>
>> GitHub: https://github.com/janl/couchdb/compare/1867-feature-plugins
>> >>>>>>>>>
>> >>>>>>>>> I created a branch on GeoCouch that adds a few
>> >>>>>>>>> lines to its `Makefile` that shows how a binary
>> >>>>>>>>> package is built:
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> https://github.com/janl/geocouch/compare/couchbase:couchdb1.3.x...couchdb1.3.x-plugins
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> * * *
>> >>>>>>>>>
>> >>>>>>>>> I hope you like this :) Please comment and improve
>> >>>>>>>>> heavily!
>> >>>>>>>>>
>> >>>>>>>>> Let me know if you have any questions :)
>> >>>>>>>>>
>> >>>>>>>>> If you have any criticism, please phrase it
in a
>> >>>>>>>>> way that we can use to improve this, thanks!
>> >>>>>>>>>
>> >>>>>>>>> Best, Jan --
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>



-- 
Andy Wenk
Hamburg - Germany
RockIt!

"CouchDB - Das Praxisbuch für Entwickler und Administratoren"
http://www.galileocomputing.de/2462
http://www.couchdb-buch.de

"PostgreSQL 8.4: Das Praxisbuch"
http://www.galileocomputing.de/2008
http://www.pg-praxisbuch.de

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: OpenPGP.js v.1.20130712
Comment: http://openpgpjs.org

xo0EUeJpHgED/34kUBUQNNT+3fcc621CLjzQZsuwYajo7Pj1hxtTcPbOo6Ci
UbyGOlIhlSDBaiXGXsFKxtdp4z/os7NdFQstzh6QpHzppjbGzGkv/om49jJM
SYLYkXyMDquhEQO47ovgOQUwJeO5qSzKE5fxftJQUjzHY1K673aA9D80uREM
Jc1tABEBAAHNF0FuZHkgV2VuayA8YW5keUBubXMuZGU+wpwEEAEIABAFAlHi
aR8JEAhxaGu1XIB2AAAfuAP/ZJXbv5wxAGCPridI/8Za9fXcccM0GmsG5ciH
bkhE9bakLlexclv3Jb+iQ2Cyp2FFo1wzLSADRRMEz1EvFFUoDo/Wj2SUQnaq
LNA8tKkRBuW0tUf88aK66TcdRINghhAcEqVJtwRIXF7fI5Arv6N+ql8heD3O
4/jA6c/8iExS0dE=
=6ftE
-----END PGP PUBLIC KEY BLOCK-----

Mime
View raw message