Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
From: Johs Ensby <johs@b2w.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_0BC25374-AAD3-44CA-91AE-08D78ABDE33D"
Message-Id: <22CAF2E9-57D9-4328-8BEF-BD0E0DAD5930@b2w.com>
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
Subject: Re: [PROPOSAL] Allow rewrites to be JS function
Date: Tue, 10 Nov 2015 10:24:39 +0100
References: 
 <CAHw0Xru9uUHM4pTDn14aQDsux-fce29GpSRcc4_LDHAiKtNekg@mail.gmail.com>
 <CACLy94W6phMQ=2XE8W7swrdKEsdbAGBeX00POyViTDuOBWD1aQ@mail.gmail.com>
 <CAHw0XrtreEWXp9=-4n2TbywwtVTd00kHjp1dVuMA80NvEg5LvQ@mail.gmail.com>
 <AC5FD406-8272-421F-824A-3892F84A3801@apache.org>
 <467C2D4D-971D-4E5C-A9AA-D3B17FD18220@b2w.com>
 <CAHdjipKOhfqVBDkVjWpP3vxZb91f-cjtU47hYNS8HLwqkx6WXQ@mail.gmail.com>
 <CADA9kr_pg8v6p0Muw-KyOhYoTwccksk0Myntwd+XRzR5Yntajw@mail.gmail.com>
 <7EB4541B-630C-47A8-A2FF-2E4B04EE8EB1@b2w.com>
 <CADA9kr9HR+atY+Yftihp-d3UTZ_-zLkutU0V+1TEUNbsvBf4ww@mail.gmail.com>
 <700C10F8-97B2-45AC-B9CA-C19DB5B36B1F@b2w.com>
 <CADA9kr-duY7HeBsaYy79d56VE6_EnyFzsckAAjunUfcp0uS7tQ@mail.gmail.com>
 <7243D1CA-C398-423D-84E2-C12FEC1A1B6D@b2w.com>
 <63D4CE75-97BC-4611-82A9-45C98753EFB5@redcometlabs.com>
 <EB4BFBE4-987C-4D1E-BCC4-0D4161D05541@b2w.com>
 <CADdZc95GtvmfEpuMCtf34bTyWxVcfCSi1Pr4UQjr7d8Vsh21Vw@mail.gmail.com>
 <CAD2UGCVAHmdKdsbcj9XgFsACq324skkTNUJmK_pq4a-1PdTP+g@mail.gmail.com>
 <C5CDDCDF-55F8-4C8E-AD28-8EC270343E5F@b2w.com>
 <CAD2UGCWUTmuTnHe1XpV+6xOV5bo9p_mRnB4oFQm9PZX_dv4H-g@mail.gmail.com>
To: "dev@couchdb.apache.org Developers" <dev@couchdb.apache.org>,
 dale@arandomurl.com
In-Reply-To: 
 <CAD2UGCWUTmuTnHe1XpV+6xOV5bo9p_mRnB4oFQm9PZX_dv4H-g@mail.gmail.com>

--Apple-Mail=_0BC25374-AAD3-44CA-91AE-08D78ABDE33D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Dale,
the discussions on Couchapps has been an endless circle, so I wanted to =
write up some thoughts before I answered this.
Also with the new mailing list couchapp@couchdb.apache.org, we will be =
able to build a discussion on this.

What you propose is clear as crystal if node.js is your platform and you =
want the fastest way to stich something together. Or you are a master of =
the npm universe and  love big puzzles.
The challenge for CouchDB if node.js is the prerequisite, is that Pouch =
server becomes the alternative to CouchDB, and MondoDB is a less risky =
choice for the pure DB. The cluster ambition seems far fetched and with =
no appeal to front-end developers, the distance to the =
revenue-generating people increase.

My idea is that with a few tweaks to the very limited 5-year-old =
application server features of CouchDB, it becomes a platform of =
integration and could be the center of a new ecosystem. At its present =
path, CouchDB is on its way to a competitive landscape where the other =
branch of the *ouch* family is challenging MongoDB, and I think it is a =
reasonable hedge for Apache CouchDB to nurture the app server features =
and differentiate as much as possible from Couchbase. If it could be =
included in the 1.7 release I would be extremely happy. I think it would =
be a very cheap insurance for the CouchDB project.

Ermouth put together a rewrite function to replace the static rewrite =
list. I have played with it on a CouchDB 2 build and it makes it =
extremely easy to do things like
- firewalls
- nested rewrite rules
- regexp based rewrites
- add a modification of your favorute client side router
- user id based rewrites
- peer IP adress based rewrites
- time based rewrite rules
- rewrite rules that make turn get requests to updates
- round robin to multiply CouchDB's request per second capacity
- etc
Of course you can do all of this better elsewhere, but can you install =
all of it over a single REST request anywhere else than on CouchDB?
I think of the design document to which you point your vhost to as an =
application server, and it will be very fun for a lot of people that are =
years away from mastering the tools to get a license to trade.=20
There is an opportunity here that could render node.js as uncool for a =
new generation of developers as the LAMP stack is for today's =
developers.

There is no downside to including ermouth's upgrade to the rewrite, and =
I believe Alexander is working on that.
Why would we not create a playground for javascript prgrammers inside =
CouchDB?
It is just an incredible long list of upsides, if anything is a =
no-brainer it is this one.

To better explain my thinking I am working on a series of articles, the =
two first here

Applications in CouchDB
- A business strategy=20
https://medium.com/p/946a9c19015 <https://medium.com/p/946a9c19015>

A possible ecosystem for CouchDB applications
https://medium.com/p/e39ac4397cea <https://medium.com/p/e39ac4397cea>

The last one ends with a plan for a pilot project that identifies 10 =
different specialized roles in a CouchDB-centric ecosystem where revenue =
flow back from the paying end user through 5 of the roles. Each role in =
a hierarchy allows them to specialize on a buseinss relationship with =
1-3 other parties.

CouchDB is at the front, PouchDB travels inside as attachment to ddocs, =
as will any js framwork, node.js is at the back as a service broker.
The main idea: create room for people at the customer-end of the value =
chain, and lay the foundation for a short multiplication cycle to =
provide some cash infusion to the CouchDB community to secure its =
future.

Johs

> On 20. okt. 2015, at 16.05, Dale Harvey <dale@arandomurl.com> wrote:
>=20
> On the server side you can run what ever JavaScript you want, it is =
far
> less limited than
> client side JavaScript. PouchDB also runs on the server and has the =
exact
> same issue
> in terms of seperation of concerns.
>=20
> PouchDB doesnt expect developers to do dom manipulation, templating,
> scheduling and
> other applications features via API's it provides, neither will it =
ever
> provide a reverse proxy
>=20
> Pouch and CouchDB can both better integrate with things that do =
provide
> reverse proxying
> and other such features much better though. Here is an example of a =
simple
> server that
> provides a proxy on /proxy, and admin party database on /db and a read =
only
> database
> on /public (it could use either pouchdb or couchdb to provide the =
databases)
>=20
> https://gist.github.com/daleharvey/bf4e9a94553b75ef2d39
>=20
> Its far from ideal, but already simpler than attempting the same setup =
with
> CouchDB only
> and far far more extendable. I didnt need to wait for CouchDB to =
implement
> any features
> or discuss their implementation.
>=20
> While CouchDB is your only application platform, there will never be =
enough
> functionality in
> it to fulfill even the basic use cases and people will lean towards =
better
> solutions
> (see mongo + meteor).
>=20
> I very much think a single shot application server is a great idea =
(see
> firebase), however
> I am very much of the opinion that it should be built on top of =
CouchDB,
> not into it.
>=20
> On 20 October 2015 at 14:37, Johs. E <johs@b2w.com> wrote:
>=20
>> Hi Dale
>> I dont understand the =E2=80=9Cintegrate with their use case as best =
as we
>> possibly can=E2=80=9D when it comes to rewrite and reverse proxy.
>> As far as I know there is nothing to integrate here.
>> The joy for PouchDB to be a pure database is very understandable, =
since it
>> sits in this rich environment that allow you to do anything.
>> At the server side CouchDB allows for a very limited portion of =
javaScript
>> processing, it would be good if it was less restrictive.
>> I am not advocating arbitrary application platform functionality.
>> Just take out some limitations for the javascript crowd out there to =
love
>> CouchDB
>>=20
>> Thanks and best regards,
>> Johs
>>=20
>>=20
>>> On 20 Oct 2015, at 11:10, Dale Harvey <dale@arandomurl.com> wrote:
>>>=20
>>> This discussion has gone round and round a couple of times in =
different
>>> forms
>>> so I will avoid repeating my previous points but from working on =
PouchDB,
>>> the focus on having 'PouchDB' be a database only is fairly =
liberating, by
>>> not trying
>>> to add fairly arbitrary "application platform" features into the =
core
>>> codebase we can focus
>>> on integrating with what does provide those application platform =
features
>>> much better.
>>>=20
>>> Users do not need to wait for reverse proxying or url rewriting to =
be an
>>> agreed, implemented
>>> and maintained feature, they can use the many that already exist and =
we
>>> will ensure
>>> that we integrate with their use case as best as we possibly can.
>>>=20
>>> On 19 October 2015 at 23:57, Harald Kisch <haraldkisch@gmail.com> =
wrote:
>>>=20
>>>> Hi Garren,
>>>>=20
>>>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying =
to
>>>> support their users with markup languages like XML, HTML, etc. for
>> instance
>>>> directly out of the database core (performance, simplicity,
>>>> scalability,..).
>>>> Lotus Notes did also integrate JavaScript inside of their core (Do =
you
>> know
>>>> which guy did take part of it?). This have different reasons, but =
one of
>>>> this reasons is to support users with dynamic mutable data directly =
into
>>>> their GUI in JSON format which in my opinion is the fundamental =
part of
>>>> CouchDB to be a database for the web.
>>>> Improvements get lost if we look at others and try not to be =
different.
>> In
>>>> my opinion we should more think about replacing spidermonkey with =
the
>>>> google V8 engine and itegrate node completely into the CouchDB core =
to
>>>> consume npm-packages directly instead of using them in the local
>> filesystem
>>>> outside of CouchDB, where unfortunatelly complexity rise up at =
scaling.
>>>>=20
>>>> --Harald
>>>>=20
>>>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby <johs@b2w.com> wrote:
>>>>=20
>>>>> Hi Garren,
>>>>> thanks for the "not standing in the way", I hope for more =
volunteers to
>>>>> iron out some of CouchDB's old akward wrinkles.
>>>>> I am all with you for simplification:) ermouth's rewrite function =
is a
>>>>> huge simplifier.
>>>>>=20
>>>>> Where I disagree with you is where you say "probably a sign that =
this
>>>> idea
>>>>> is not something worth pursuing".
>>>>> Whenever you discover that you have a differentiator, it's always =
a
>> good
>>>>> idea to look closely before discaring it and blend in with the =
rest.
>>>>> It's all about attracting the next million web developers.
>>>>>=20
>>>>> johs
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>> On 19. okt. 2015, at 20.08, Garren Smith <gs@redcometlabs.com> =
wrote:
>>>>>>=20
>>>>>> I=E2=80=99m really struggling with these proposals. I love the =
enthusiasm of
>>>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>>>> CouchDB is ultimately a database. One with excellent sync
>> capabilities.
>>>>> And combining that with libraries like PouchDB and Hoodie make it =
an
>>>>> amazing database to build applications with.
>>>>>> Adding routers and reverse proxies just makes it feel like we =
trying
>> to
>>>>> push CouchDB into being more than it needs to be.
>>>>>>=20
>>>>>> For example building Couchapp like functionality in Node.js is so
>>>> simple
>>>>> and way better. Languages like Go also do that really well. Far
>> superior
>>>>> than what we can do with a database.
>>>>>> I would rather let the Node.js and Go web libraries serve content =
and
>>>>> let us focus on building a clustered replicating database. We will =
draw
>>>>> more people to this community if we can do that properly over =
creaky,
>>>> slow
>>>>> and limited web serving mashed on top of a database.
>>>>>> If I look at other popular databases, I don=E2=80=99t see any of =
them serving
>>>>> web content which is probably a sign that this idea is not =
something
>>>> worth
>>>>> pursuing.
>>>>>>=20
>>>>>> However if there is a burning desire for this and developers =
raising
>>>>> their hands to code this functionality, I would not stand in your =
way.
>> It
>>>>> is great to see the varied use of CouchDB out in the wild.
>>>>>>=20
>>>>>> Cheers
>>>>>> Garren
>>>>>>=20
>>>>>>=20
>>>>>>> On 19 Oct 2015, at 4:47 PM, Johs. E <johs@b2w.com> wrote:
>>>>>>>=20
>>>>>>> Thanks Andy,
>>>>>>> I will try and  get some use cases up on confluence.
>>>>>>> As for whoever would pick up the work after ermouth,  I have of
>> course
>>>>> one big thing on the wish list that goes well with a new router
>>>> solution..
>>>>>>> reverse proxy
>>>>>>> I remember asking about it when I first started to work w =
CouchDB and
>>>>> there were some concerns regarding security.
>>>>>>> Since then I think node.js has paved the way with content =
scraping
>> and
>>>>> all sorts of outgoing traffic.
>>>>>>>=20
>>>>>>> Has anyone work on a reverse proxy solution for Couch?
>>>>>>>=20
>>>>>>> johs
>>>>>>>=20
>>>>>>>=20
>>>>>>>> On 18 Oct 2015, at 21:36, Andy Wenk <andywenk@apache.org> =
wrote:
>>>>>>>>=20
>>>>>>>> Hey Johs,
>>>>>>>>=20
>>>>>>>> thanks a lot for this. I need some time to dig into it. We need =
a
>>>>> place to
>>>>>>>> write the user stories / use case down. So I suggest we find =
good
>>>>> place at
>>>>>>>> the cwiki. So I suggest to use
>>>>>>>>=20
>>>>>=20
>>>>=20
>> =
https://cwiki.apache.org/confluence/display/COUCHDB/Enhancement+Proposals
>>>>> .
>>>>>>>> Do you have write access there? If not, please ping me.
>>>>>>>>=20
>>>>>>>> Great work!
>>>>>>>>=20
>>>>>>>> All the best
>>>>>>>>=20
>>>>>>>> Andy
>>>>>>>>=20
>>>>>>>> P.S.: Jan already mentioned the feature freeze. Please take it =
not
>>>> as a
>>>>>>>> demotivation but as the possibility to have a bit more time to =
work
>>>> on
>>>>> it.
>>>>>>>>=20
>>>>>>>> On 17 October 2015 at 17:32, Johs Ensby <johs@b2w.com> wrote:
>>>>>>>>=20
>>>>>>>>> Andy,
>>>>>>>>> I will make my first use case for function in _rewrite a high =
level
>>>>> one:
>>>>>>>>> to create a standalone server that is an all-in-one DB server,
>>>>> application
>>>>>>>>> server, api server and web server.
>>>>>>>>>=20
>>>>>>>>> I have played with the build of CouchDB 2 with rewrite =
function
>>>>>>>>> implemented that  ermouth put up on the irish AWS community =
AMI
>> list
>>>>> and
>>>>>>>>> the new use cases are endless.
>>>>>>>>> First, I find that there are a few things that people fail to
>> notice
>>>>> about
>>>>>>>>> ddocs.
>>>>>>>>> you need a tool to build a ddoc, editing JSON is not a viable
>>>> option.
>>>>> The
>>>>>>>>> Ddoc Lab of ermouth is in a class of its own. If you havent =
tried
>> it
>>>>> yet,
>>>>>>>>> do so from http://ddoc.me/ <http://ddoc.me/>. Installing on =
your
>>>> own
>>>>>>>>> couch it is as easy as storing the application, all included =
as one
>>>>>>>>> document in any database. Ddoc Lab is a component oriented IDE =
with
>>>>> syntax
>>>>>>>>> checking, less preprosessor and other build tools that let you =
keep
>>>> a
>>>>> well
>>>>>>>>> organized ddoc as a source project (in one couchdb document) =
and
>> you
>>>>>>>>> publich a ddoc to any target db.
>>>>>>>>> with this tool you can organize your js modules and templates =
etc
>>>> and
>>>>>>>>> basically...
>>>>>>>>> set up the API of your application in a ddoc. You can switch
>> between
>>>>>>>>> databases and their ddoc functionality based on username, role =
or
>>>>>>>>> geolocation and limit access to parts of the Couch API as =
needed
>>>>>>>>>=20
>>>>>>>>> This is the method I would recommend to explore powerful =
simplicity
>>>>> with
>>>>>>>>> function in rewrites
>>>>>>>>> redirect port 80 directly to couch
>>>>>>>>> set up 2 vhosts, one for public access pointing to
>> youdb/_design/api
>>>>> and
>>>>>>>>> one for sysadm pointing to /
>>>>>>>>> for admin use Fauxton and Ddoc Lab on the sysadm vhost
>>>>>>>>> you are set to develop great systems, no big tool stack to =
learn,
>>>> just
>>>>>>>>> bring in whatever js modules you like, the template engine you
>> like,
>>>>> the
>>>>>>>>> router you like, the HTML5 stuff you like..
>>>>>>>>> .. or just write some very compact js code in one place where =
you
>>>>> ealier
>>>>>>>>> had to mess around with a whole stack of tools and systems
>>>>>>>>>=20
>>>>>>>>> below is the req object that the function takes
>>>>>>>>>=20
>>>>>>>>> Johs
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> The rewrite function has this syntax
>>>>>>>>> function(req) {
>>>>>>>>>    .. your code that will
>>>>>>>>>    return
>>>>>>>>>            path
>>>>>>>>>            // optional
>>>>>>>>>            headers
>>>>>>>>>            method // you can change this on the fly
>>>>>>>>>            code
>>>>>>>>>            body
>>>>>>>>> }
>>>>>>>>>=20
>>>>>>>>> the function receives this req object
>>>>>>>>> method
>>>>>>>>> path
>>>>>>>>> raw_path
>>>>>>>>> query
>>>>>>>>> headers
>>>>>>>>>    Accept
>>>>>>>>>    Accept-Encoding
>>>>>>>>>    Connection
>>>>>>>>>    Host
>>>>>>>>>    Upgrade-Insecure-Requests
>>>>>>>>>    User-Agent
>>>>>>>>>    x-couchdb-vhost-path
>>>>>>>>> body
>>>>>>>>> peer
>>>>>>>>> cookie
>>>>>>>>> userCtx
>>>>>>>>> db
>>>>>>>>> name
>>>>>>>>> roles
>>>>>>>>> secObj
>>>>>>>>>=20
>>>>>>>>>> On 1. okt. 2015, at 13.40, Andy Wenk <andywenk@apache.org> =
wrote:
>>>>>>>>>>=20
>>>>>>>>>> Johs,
>>>>>>>>>>=20
>>>>>>>>>> Yes for sure! That's always great. Maybe you can also write =
some
>>>> user
>>>>>>>>> stories (given when then) or scribble some graphics. =
Everything is
>>>>> useful
>>>>>>>>> and will fasten the process ;-)
>>>>>>>>>>=20
>>>>>>>>>> Cheers
>>>>>>>>>>=20
>>>>>>>>>> Andy
>>>>>>>>>>=20
>>>>>>>>>> On 1 Oct 2015 12:38, "Johs Ensby" <johs@b2w.com <mailto:
>>>> johs@b2w.com
>>>>>>>=20
>>>>>>>>> wrote:
>>>>>>>>>> Thanks for this Andy,
>>>>>>>>>>=20
>>>>>>>>>> I can contribute to the discussion of the feature seen from a =
user
>>>>>>>>> perspective.
>>>>>>>>>> Would it be appropriate to present some use cases?
>>>>>>>>>>=20
>>>>>>>>>> best
>>>>>>>>>> Johs
>>>>>>>>>>=20
>>>>>>>>>>> On 1. okt. 2015, at 12.33, Andy Wenk <andywenk@apache.org
>>>> <mailto:
>>>>>>>>> andywenk@apache.org>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>> Johs,
>>>>>>>>>>>=20
>>>>>>>>>>> Let me please show the steps needed.
>>>>>>>>>>>=20
>>>>>>>>>>> * discuss the feature very clearly on the dev@. Please make =
sure
>>>>> that
>>>>>>>>> core
>>>>>>>>>>> developers as committers with commit bits are involved
>>>>>>>>>>>=20
>>>>>>>>>>> * code the feature. Make sure to implement tests
>>>>>>>>>>>=20
>>>>>>>>>>> * send a pull request and show it to dev@
>>>>>>>>>>>=20
>>>>>>>>>>> * finally the community will accept or decline the feature =
(this
>>>>> will
>>>>>>>>>>> involve refactoring and changes)
>>>>>>>>>>>=20
>>>>>>>>>>> As Alex said. The PMC or Jan do not decide about the =
feature.
>>>>>>>>>>>=20
>>>>>>>>>>> All the best
>>>>>>>>>>>=20
>>>>>>>>>>> Andy
>>>>>>>>>>> On 1 Oct 2015 11:21, "Alexander Shorin" <kxepal@gmail.com
>>>> <mailto:
>>>>>>>>> kxepal@gmail.com>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> On Thu, Oct 1, 2015 at 12:07 PM, Johs Ensby <johs@b2w.com
>>>> <mailto:
>>>>>>>>> johs@b2w.com>> wrote:
>>>>>>>>>>>>> will you welcome ermouths rewrite contribution?
>>>>>>>>>>>>=20
>>>>>>>>>>>> The decision is depends on the implementation. If it will =
be
>>>> good,
>>>>> why
>>>>>>>>>>>> not? Finally, CouchDB is open source project: we cannot =
forbid
>>>>> people
>>>>>>>>>>>> right for contributions, we only welcome them.
>>>>>>>>>>>>=20
>>>>>>>>>>>>> Arguments against couchapps has to do with performance and =
the
>>>>> folly
>>>>>>>>> in
>>>>>>>>>>>> competing with node.js.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Performance question for the new _rewrite implementation is =
very
>>>>>>>>>>>> depends on query server. Once it can process this kind of
>>>>> functions,
>>>>>>>>>>>> you may use something better than JS to gain better =
performance.
>>>>> That
>>>>>>>>>>>> could be Erlang native query server, or luerl-based one, or =
else
>>>>> you
>>>>>>>>>>>> like.
>>>>>>>>>>>>=20
>>>>>>>>>>>> --
>>>>>>>>>>>> ,,,^..^,,,
>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> --
>>>>>>>> Andy Wenk
>>>>>>>> Hamburg - Germany
>>>>>>>> RockIt!
>>>>>>>>=20
>>>>>>>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 =
9588
>>>>>>>>=20
>>>>>>>> https://people.apache.org/keys/committer/andywenk.asc
>>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>=20
>>=20
>>=20


--Apple-Mail=_0BC25374-AAD3-44CA-91AE-08D78ABDE33D--