couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johs Ensby <j...@b2w.com>
Subject Re: [PROPOSAL] Allow rewrites to be JS function
Date Tue, 10 Nov 2015 09:24:39 GMT
Hi Dale,
the discussions on Couchapps has been an endless circle, so I wanted to write up some thoughts
before I answered this.
Also with the new mailing list couchapp@couchdb.apache.org, we will be able to build a discussion
on this.

What you propose is clear as crystal if node.js is your platform and you want the fastest
way to stich something together. Or you are a master of the npm universe and  love big puzzles.
The challenge for CouchDB if node.js is the prerequisite, is that Pouch server becomes the
alternative to CouchDB, and MondoDB is a less risky choice for the pure DB. The cluster ambition
seems far fetched and with no appeal to front-end developers, the distance to the revenue-generating
people increase.

My idea is that with a few tweaks to the very limited 5-year-old application server features
of CouchDB, it becomes a platform of integration and could be the center of a new ecosystem.
At its present path, CouchDB is on its way to a competitive landscape where the other branch
of the *ouch* family is challenging MongoDB, and I think it is a reasonable hedge for Apache
CouchDB to nurture the app server features and differentiate as much as possible from Couchbase.
If it could be included in the 1.7 release I would be extremely happy. I think it would be
a very cheap insurance for the CouchDB project.

Ermouth put together a rewrite function to replace the static rewrite list. I have played
with it on a CouchDB 2 build and it makes it extremely easy to do things like
- firewalls
- nested rewrite rules
- regexp based rewrites
- add a modification of your favorute client side router
- user id based rewrites
- peer IP adress based rewrites
- time based rewrite rules
- rewrite rules that make turn get requests to updates
- round robin to multiply CouchDB's request per second capacity
- etc
Of course you can do all of this better elsewhere, but can you install all of it over a single
REST request anywhere else than on CouchDB?
I think of the design document to which you point your vhost to as an application server,
and it will be very fun for a lot of people that are years away from mastering the tools to
get a license to trade. 
There is an opportunity here that could render node.js as uncool for a new generation of developers
as the LAMP stack is for today's developers.

There is no downside to including ermouth's upgrade to the rewrite, and I believe Alexander
is working on that.
Why would we not create a playground for javascript prgrammers inside CouchDB?
It is just an incredible long list of upsides, if anything is a no-brainer it is this one.

To better explain my thinking I am working on a series of articles, the two first here

Applications in CouchDB
- A business strategy 
https://medium.com/p/946a9c19015 <https://medium.com/p/946a9c19015>

A possible ecosystem for CouchDB applications
https://medium.com/p/e39ac4397cea <https://medium.com/p/e39ac4397cea>

The last one ends with a plan for a pilot project that identifies 10 different specialized
roles in a CouchDB-centric ecosystem where revenue flow back from the paying end user through
5 of the roles. Each role in a hierarchy allows them to specialize on a buseinss relationship
with 1-3 other parties.

CouchDB is at the front, PouchDB travels inside as attachment to ddocs, as will any js framwork,
node.js is at the back as a service broker.
The main idea: create room for people at the customer-end of the value chain, and lay the
foundation for a short multiplication cycle to provide some cash infusion to the CouchDB community
to secure its future.

Johs

> On 20. okt. 2015, at 16.05, Dale Harvey <dale@arandomurl.com> wrote:
> 
> On the server side you can run what ever JavaScript you want, it is far
> less limited than
> client side JavaScript. PouchDB also runs on the server and has the exact
> same issue
> in terms of seperation of concerns.
> 
> PouchDB doesnt expect developers to do dom manipulation, templating,
> scheduling and
> other applications features via API's it provides, neither will it ever
> provide a reverse proxy
> 
> Pouch and CouchDB can both better integrate with things that do provide
> reverse proxying
> and other such features much better though. Here is an example of a simple
> server that
> provides a proxy on /proxy, and admin party database on /db and a read only
> database
> on /public (it could use either pouchdb or couchdb to provide the databases)
> 
> https://gist.github.com/daleharvey/bf4e9a94553b75ef2d39
> 
> Its far from ideal, but already simpler than attempting the same setup with
> CouchDB only
> and far far more extendable. I didnt need to wait for CouchDB to implement
> any features
> or discuss their implementation.
> 
> While CouchDB is your only application platform, there will never be enough
> functionality in
> it to fulfill even the basic use cases and people will lean towards better
> solutions
> (see mongo + meteor).
> 
> I very much think a single shot application server is a great idea (see
> firebase), however
> I am very much of the opinion that it should be built on top of CouchDB,
> not into it.
> 
> On 20 October 2015 at 14:37, Johs. E <johs@b2w.com> wrote:
> 
>> Hi Dale
>> I dont understand the “integrate with their use case as best as we
>> possibly can” when it comes to rewrite and reverse proxy.
>> As far as I know there is nothing to integrate here.
>> The joy for PouchDB to be a pure database is very understandable, since it
>> sits in this rich environment that allow you to do anything.
>> At the server side CouchDB allows for a very limited portion of javaScript
>> processing, it would be good if it was less restrictive.
>> I am not advocating arbitrary application platform functionality.
>> Just take out some limitations for the javascript crowd out there to love
>> CouchDB
>> 
>> Thanks and best regards,
>> Johs
>> 
>> 
>>> On 20 Oct 2015, at 11:10, Dale Harvey <dale@arandomurl.com> wrote:
>>> 
>>> This discussion has gone round and round a couple of times in different
>>> forms
>>> so I will avoid repeating my previous points but from working on PouchDB,
>>> the focus on having 'PouchDB' be a database only is fairly liberating, by
>>> not trying
>>> to add fairly arbitrary "application platform" features into the core
>>> codebase we can focus
>>> on integrating with what does provide those application platform features
>>> much better.
>>> 
>>> Users do not need to wait for reverse proxying or url rewriting to be an
>>> agreed, implemented
>>> and maintained feature, they can use the many that already exist and we
>>> will ensure
>>> that we integrate with their use case as best as we possibly can.
>>> 
>>> On 19 October 2015 at 23:57, Harald Kisch <haraldkisch@gmail.com> wrote:
>>> 
>>>> Hi Garren,
>>>> 
>>>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying to
>>>> support their users with markup languages like XML, HTML, etc. for
>> instance
>>>> directly out of the database core (performance, simplicity,
>>>> scalability,..).
>>>> Lotus Notes did also integrate JavaScript inside of their core (Do you
>> know
>>>> which guy did take part of it?). This have different reasons, but one of
>>>> this reasons is to support users with dynamic mutable data directly into
>>>> their GUI in JSON format which in my opinion is the fundamental part of
>>>> CouchDB to be a database for the web.
>>>> Improvements get lost if we look at others and try not to be different.
>> In
>>>> my opinion we should more think about replacing spidermonkey with the
>>>> google V8 engine and itegrate node completely into the CouchDB core to
>>>> consume npm-packages directly instead of using them in the local
>> filesystem
>>>> outside of CouchDB, where unfortunatelly complexity rise up at scaling.
>>>> 
>>>> --Harald
>>>> 
>>>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby <johs@b2w.com> wrote:
>>>> 
>>>>> Hi Garren,
>>>>> thanks for the "not standing in the way", I hope for more volunteers
to
>>>>> iron out some of CouchDB's old akward wrinkles.
>>>>> I am all with you for simplification:) ermouth's rewrite function is
a
>>>>> huge simplifier.
>>>>> 
>>>>> Where I disagree with you is where you say "probably a sign that this
>>>> idea
>>>>> is not something worth pursuing".
>>>>> Whenever you discover that you have a differentiator, it's always a
>> good
>>>>> idea to look closely before discaring it and blend in with the rest.
>>>>> It's all about attracting the next million web developers.
>>>>> 
>>>>> johs
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 19. okt. 2015, at 20.08, Garren Smith <gs@redcometlabs.com>
wrote:
>>>>>> 
>>>>>> I’m really struggling with these proposals. I love the enthusiasm
of
>>>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>>>> CouchDB is ultimately a database. One with excellent sync
>> capabilities.
>>>>> And combining that with libraries like PouchDB and Hoodie make it an
>>>>> amazing database to build applications with.
>>>>>> Adding routers and reverse proxies just makes it feel like we trying
>> to
>>>>> push CouchDB into being more than it needs to be.
>>>>>> 
>>>>>> For example building Couchapp like functionality in Node.js is so
>>>> simple
>>>>> and way better. Languages like Go also do that really well. Far
>> superior
>>>>> than what we can do with a database.
>>>>>> I would rather let the Node.js and Go web libraries serve content
and
>>>>> let us focus on building a clustered replicating database. We will draw
>>>>> more people to this community if we can do that properly over creaky,
>>>> slow
>>>>> and limited web serving mashed on top of a database.
>>>>>> If I look at other popular databases, I don’t see any of them serving
>>>>> web content which is probably a sign that this idea is not something
>>>> worth
>>>>> pursuing.
>>>>>> 
>>>>>> However if there is a burning desire for this and developers raising
>>>>> their hands to code this functionality, I would not stand in your way.
>> It
>>>>> is great to see the varied use of CouchDB out in the wild.
>>>>>> 
>>>>>> Cheers
>>>>>> Garren
>>>>>> 
>>>>>> 
>>>>>>> On 19 Oct 2015, at 4:47 PM, Johs. E <johs@b2w.com> wrote:
>>>>>>> 
>>>>>>> Thanks Andy,
>>>>>>> I will try and  get some use cases up on confluence.
>>>>>>> As for whoever would pick up the work after ermouth,  I have
of
>> course
>>>>> one big thing on the wish list that goes well with a new router
>>>> solution..
>>>>>>> reverse proxy
>>>>>>> I remember asking about it when I first started to work w CouchDB
and
>>>>> there were some concerns regarding security.
>>>>>>> Since then I think node.js has paved the way with content scraping
>> and
>>>>> all sorts of outgoing traffic.
>>>>>>> 
>>>>>>> Has anyone work on a reverse proxy solution for Couch?
>>>>>>> 
>>>>>>> johs
>>>>>>> 
>>>>>>> 
>>>>>>>> On 18 Oct 2015, at 21:36, Andy Wenk <andywenk@apache.org>
wrote:
>>>>>>>> 
>>>>>>>> Hey Johs,
>>>>>>>> 
>>>>>>>> thanks a lot for this. I need some time to dig into it. We
need a
>>>>> place to
>>>>>>>> write the user stories / use case down. So I suggest we find
good
>>>>> place at
>>>>>>>> the cwiki. So I suggest to use
>>>>>>>> 
>>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/COUCHDB/Enhancement+Proposals
>>>>> .
>>>>>>>> Do you have write access there? If not, please ping me.
>>>>>>>> 
>>>>>>>> Great work!
>>>>>>>> 
>>>>>>>> All the best
>>>>>>>> 
>>>>>>>> Andy
>>>>>>>> 
>>>>>>>> P.S.: Jan already mentioned the feature freeze. Please take
it not
>>>> as a
>>>>>>>> demotivation but as the possibility to have a bit more time
to work
>>>> on
>>>>> it.
>>>>>>>> 
>>>>>>>> On 17 October 2015 at 17:32, Johs Ensby <johs@b2w.com>
wrote:
>>>>>>>> 
>>>>>>>>> Andy,
>>>>>>>>> I will make my first use case for function in _rewrite
a high level
>>>>> one:
>>>>>>>>> to create a standalone server that is an all-in-one DB
server,
>>>>> application
>>>>>>>>> server, api server and web server.
>>>>>>>>> 
>>>>>>>>> I have played with the build of CouchDB 2 with rewrite
function
>>>>>>>>> implemented that  ermouth put up on the irish AWS community
AMI
>> list
>>>>> and
>>>>>>>>> the new use cases are endless.
>>>>>>>>> First, I find that there are a few things that people
fail to
>> notice
>>>>> about
>>>>>>>>> ddocs.
>>>>>>>>> you need a tool to build a ddoc, editing JSON is not
a viable
>>>> option.
>>>>> The
>>>>>>>>> Ddoc Lab of ermouth is in a class of its own. If you
havent tried
>> it
>>>>> yet,
>>>>>>>>> do so from http://ddoc.me/ <http://ddoc.me/>. Installing
on your
>>>> own
>>>>>>>>> couch it is as easy as storing the application, all included
as one
>>>>>>>>> document in any database. Ddoc Lab is a component oriented
IDE with
>>>>> syntax
>>>>>>>>> checking, less preprosessor and other build tools that
let you keep
>>>> a
>>>>> well
>>>>>>>>> organized ddoc as a source project (in one couchdb document)
and
>> you
>>>>>>>>> publich a ddoc to any target db.
>>>>>>>>> with this tool you can organize your js modules and templates
etc
>>>> and
>>>>>>>>> basically...
>>>>>>>>> set up the API of your application in a ddoc. You can
switch
>> between
>>>>>>>>> databases and their ddoc functionality based on username,
role or
>>>>>>>>> geolocation and limit access to parts of the Couch API
as needed
>>>>>>>>> 
>>>>>>>>> This is the method I would recommend to explore powerful
simplicity
>>>>> with
>>>>>>>>> function in rewrites
>>>>>>>>> redirect port 80 directly to couch
>>>>>>>>> set up 2 vhosts, one for public access pointing to
>> youdb/_design/api
>>>>> and
>>>>>>>>> one for sysadm pointing to /
>>>>>>>>> for admin use Fauxton and Ddoc Lab on the sysadm vhost
>>>>>>>>> you are set to develop great systems, no big tool stack
to learn,
>>>> just
>>>>>>>>> bring in whatever js modules you like, the template engine
you
>> like,
>>>>> the
>>>>>>>>> router you like, the HTML5 stuff you like..
>>>>>>>>> .. or just write some very compact js code in one place
where you
>>>>> ealier
>>>>>>>>> had to mess around with a whole stack of tools and systems
>>>>>>>>> 
>>>>>>>>> below is the req object that the function takes
>>>>>>>>> 
>>>>>>>>> Johs
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> The rewrite function has this syntax
>>>>>>>>> function(req) {
>>>>>>>>>    .. your code that will
>>>>>>>>>    return
>>>>>>>>>            path
>>>>>>>>>            // optional
>>>>>>>>>            headers
>>>>>>>>>            method // you can change this on the fly
>>>>>>>>>            code
>>>>>>>>>            body
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> the function receives this req object
>>>>>>>>> method
>>>>>>>>> path
>>>>>>>>> raw_path
>>>>>>>>> query
>>>>>>>>> headers
>>>>>>>>>    Accept
>>>>>>>>>    Accept-Encoding
>>>>>>>>>    Connection
>>>>>>>>>    Host
>>>>>>>>>    Upgrade-Insecure-Requests
>>>>>>>>>    User-Agent
>>>>>>>>>    x-couchdb-vhost-path
>>>>>>>>> body
>>>>>>>>> peer
>>>>>>>>> cookie
>>>>>>>>> userCtx
>>>>>>>>> db
>>>>>>>>> name
>>>>>>>>> roles
>>>>>>>>> secObj
>>>>>>>>> 
>>>>>>>>>> On 1. okt. 2015, at 13.40, Andy Wenk <andywenk@apache.org>
wrote:
>>>>>>>>>> 
>>>>>>>>>> Johs,
>>>>>>>>>> 
>>>>>>>>>> Yes for sure! That's always great. Maybe you can
also write some
>>>> user
>>>>>>>>> stories (given when then) or scribble some graphics.
Everything is
>>>>> useful
>>>>>>>>> and will fasten the process ;-)
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> 
>>>>>>>>>> Andy
>>>>>>>>>> 
>>>>>>>>>> On 1 Oct 2015 12:38, "Johs Ensby" <johs@b2w.com
<mailto:
>>>> johs@b2w.com
>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> Thanks for this Andy,
>>>>>>>>>> 
>>>>>>>>>> I can contribute to the discussion of the feature
seen from a user
>>>>>>>>> perspective.
>>>>>>>>>> Would it be appropriate to present some use cases?
>>>>>>>>>> 
>>>>>>>>>> best
>>>>>>>>>> Johs
>>>>>>>>>> 
>>>>>>>>>>> On 1. okt. 2015, at 12.33, Andy Wenk <andywenk@apache.org
>>>> <mailto:
>>>>>>>>> andywenk@apache.org>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Johs,
>>>>>>>>>>> 
>>>>>>>>>>> Let me please show the steps needed.
>>>>>>>>>>> 
>>>>>>>>>>> * discuss the feature very clearly on the dev@.
Please make sure
>>>>> that
>>>>>>>>> core
>>>>>>>>>>> developers as committers with commit bits are
involved
>>>>>>>>>>> 
>>>>>>>>>>> * code the feature. Make sure to implement tests
>>>>>>>>>>> 
>>>>>>>>>>> * send a pull request and show it to dev@
>>>>>>>>>>> 
>>>>>>>>>>> * finally the community will accept or decline
the feature (this
>>>>> will
>>>>>>>>>>> involve refactoring and changes)
>>>>>>>>>>> 
>>>>>>>>>>> As Alex said. The PMC or Jan do not decide about
the feature.
>>>>>>>>>>> 
>>>>>>>>>>> All the best
>>>>>>>>>>> 
>>>>>>>>>>> Andy
>>>>>>>>>>> On 1 Oct 2015 11:21, "Alexander Shorin" <kxepal@gmail.com
>>>> <mailto:
>>>>>>>>> kxepal@gmail.com>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Oct 1, 2015 at 12:07 PM, Johs Ensby
<johs@b2w.com
>>>> <mailto:
>>>>>>>>> johs@b2w.com>> wrote:
>>>>>>>>>>>>> will you welcome ermouths rewrite contribution?
>>>>>>>>>>>> 
>>>>>>>>>>>> The decision is depends on the implementation.
If it will be
>>>> good,
>>>>> why
>>>>>>>>>>>> not? Finally, CouchDB is open source project:
we cannot forbid
>>>>> people
>>>>>>>>>>>> right for contributions, we only welcome
them.
>>>>>>>>>>>> 
>>>>>>>>>>>>> Arguments against couchapps has to do
with performance and the
>>>>> folly
>>>>>>>>> in
>>>>>>>>>>>> competing with node.js.
>>>>>>>>>>>> 
>>>>>>>>>>>> Performance question for the new _rewrite
implementation is very
>>>>>>>>>>>> depends on query server. Once it can process
this kind of
>>>>> functions,
>>>>>>>>>>>> you may use something better than JS to gain
better performance.
>>>>> That
>>>>>>>>>>>> could be Erlang native query server, or luerl-based
one, or else
>>>>> you
>>>>>>>>>>>> like.
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> ,,,^..^,,,
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Andy Wenk
>>>>>>>> Hamburg - Germany
>>>>>>>> RockIt!
>>>>>>>> 
>>>>>>>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3
9ED3 9588
>>>>>>>> 
>>>>>>>> https://people.apache.org/keys/committer/andywenk.asc
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message