couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <>
Subject Re: [PROPOSAL] Allow rewrites to be JS function
Date Wed, 21 Oct 2015 11:58:59 GMT

> On 21 Oct 2015, at 13:43, Alexander Shorin <> wrote:
> This will never end...
> First, this thread goes to be offtopic with unrelated flame discussion
> about anything, but not new shiny JS rewrites. If you want to
> continue, please start new thread - this will help to sort all the
> things.
> Second, let's stop call things broken, dead end, doomed, without solid
> references. If we have things broken - let's file issues about. If
> things comes to the dead end, let's discuss how we can get them out of
> it , if our features has some problems - let's find solution for them.
> JIRA is a good place to accumulate all the bugs and missed features,
> ideas and proposals could be widely explained in CWiki.

I fear the only people who can do this properly are currently wrapped up
in shipping 2.0 and I’d like to not distract them from that.

The tl;dr here is what I hinted at in my earlier mail: couchjs uses a
fork-per-concurrent-request model. This only works with very low volume
applications unless you have unlimited hardware, and that’s not something
that fits in the philosophy of CouchDB where things work with low volume
requests as well as high volume requests. And even if we could get a better
pre-fork model, we still are in a synchronous execution mode of an async
language, and before we custom-fix all that, we might as well use something
that already exists, update our protocols (details in the GSoC submission
about rewriting the view server protocol that I wrote earlier this year).

To make this better, I repeatedly proposed switching our JS stack to that
other JS stack that has become popular in the past five years, but I’ve
gotten flak for that as well. For now, I’ve concluded that shipping 2.0
is the one thing we need to care about and it seems that most people here
agree, and most end-users too. Imagine all the PouchDB users rejoicing of
2.0 being available with _bulk_get/_revs, alone!

If you’d like a first-hand account what kind of things we’d enable there,
look no further than here, where CouchDB/PouchDB helped fight Ebola:

Let’s not get bogged down, again, by a niche-feature set that not a lot
of people care about.

> Third, (that would short list), let's calm down and stop speak in
> absolute or ultimate way as like as involve emotions to the
> discussion. All this will never help to find a middle ground.
> Fourth, (ok, I lied), I almost get ready to take responsibility for
> maintaining CouchApp feature and related stuff, but so far in all
> recent the discussions that happened here and on marketing@ about I
> didn't found no well formed proposal what to do except this one. How
> can we fix the feature if there is no strait forward plan for it?

To do any project you need a plan, time and people to do it with.
We have neither, except maybe some people who want to work with it,
but to come up with a plan, we’d have to take time away from people
who are shipping 2.0 (like you), and I wouldn’t like to see 2.0 get
delayed another year because we go out on a tangent here.

We absolutely don’t have to please everyone, and the biggest failing
we’ve had is not shipping 2.0 last year, so let’s rally together and
get that out the door and then see what’s left.

If the people who want to work on this want to work on this now, I
repeat that I’m happy to help set up a safe space for them, so we can
ship 2.0.


> There is no any strategy for CouchApps future by now except "fix known
> bugs". Without strategy, without plan there is no feature that can
> evolve effectively. It seems like that the most interested people in
> CouchApp feature are not CouchDB devs, but users community. So:
>    Dear, Community.
>    We have JIRA for bugs and features, we have CWiki for proposals,
> RFC and finally we're the devs. You have ideas, needs, use cases,
> practice and the strong will.
>    Let's cooperate!
> Because until this JS rewrites proposal there was only meaningless
> talks. Ermouth did all right that he started with the something and
> even made propotype of his ideas. It's sad that we have a quite
> complicated testing routines, but I'm happy to help finish his feature
> and made PR for the final discussion. That's how I see we can work.
> But to do this, we need to stop dream about far future, stop waste our
> time in endless talks "what couchapps is and what their future" and
> start build that future right now.
> Useful link for you all:
> -
> -
> (if you need access bit, please ask with your cwiki login)
> Thanks! And may the force be with you!
> --
> ,,,^..^,,,
> On Wed, Oct 21, 2015 at 2:08 PM, Harald Kisch <> wrote:
>>> This project doesn’t have the capacity to get a distributed syncing
>> database AND an application server platform going > at the same time.
>> So please explain: Why CouchAppers get bothered every time when CouchApps
>> comes to discussion?
>> I thought the CouchApp discussions are very welcome @dev. Is'nt it the fact
>> that this mailing list is exactly for this kind of discussions? Is'nt it
>> the fact that this negative emails always say: "Do not work on this."
>> Something in me is crying when I read something like that because until
>> now, I did'nt read any hard fact WHY NOT?
>> Here the dog bites in his own tail: "Why couchappers are not allowed to
>> work on CouchApps in between the CouchDB Community anymore and at the same
>> time the argument rises that it was not touched since 5 years"?
>> As far as I know, there was no request and no claim made for extending
>> CouchApp features, but I feel compromized and confronted with very bad
>> energy when it comes to this topic. I think there are many developers who
>> would love to start coding on CouchApps but not under the hood of current
>> circumstances with getting bothered every time a mail appears @marketing or
>> @dev concerning CouchApps.
>> I do not understand where this negative emotions against CouchApps comes
>> from. As mentioned before: There are people doing business on top of
>> CouchApps. It can't be buggy, incomplete, insecure or something else when
>> people start to have business on top of it. Instead of listening to them,
>> which solutions they have found, they get botherd. Could you explain
>> please, what is the reason for this emotional emails? If there is a real
>> hard fact not using CouchApps, I promise, I will stop writing emails
>> regarding CouchApps immediatly for any CouchDB mailing list.
>> --Harald
>> I would say, many people lost their interest in CouchDB because exactly for
>> such claims like:
>>> *
>>> **
>> On Tue, Oct 20, 2015 at 9:09 PM, Jan Lehnardt <> wrote:
>>> What Dale and the others said.
>>> And what I explained at length on the marketing@ mailing list to the very
>>> same people:
>>> This project doesn’t have the capacity to get a distributed syncing
>>> database AND an application server platform going at the same time. And the
>>> people who are doing the lion’s share of the work here have long concluded
>>> that we need to get the database part going before we can tackle anything
>>> else.
>>> Like others said, CouchDB is flexible enough to let anyone build this on
>>> top of, or make a plugin, or whatever, we can even make it an ASF
>>> sub-project, but this needs to happen separately from what we’ve all agreed
>>> in 2012 and again in 2014 and 15 to do here, which is ship a kick-ass
>>> database, that replicates.
>>> The one “extra-curricular” thing we have to allow ourselves is pushing the
>>> sync protocol forward, together with PouchDB & friends, and accommodate the
>>> new use cases that our syncing cousins enable for our end-users.
>>> Things like selective sync, a less chatty sync protocol for mobile users
>>> (_bulk_get/_revs is only a small patch for a larger problem). These are the
>>> pressing issues for this project that we have to get right before we can
>>> consider building an app server on top of it. And we have to ship the 2.0
>>> cluster first, too, so anyone with JavaScript chops here should join the
>>> “The JavaScript Test Suite” thread and get hacking.
>>> If you are interested in building an app server on top of CouchDB, do
>>> that: build it on top of CouchDB. We are more than happy put you into the
>>> Apache family and help with naming, community, security, distribution etc.
>>> issues, but this won’t be the core focus of this project.
>>> And like I mentioned to you very same people on marketing@ before*,**:
>>> I’m tired of the deliberate derailing of this project.
>>> It stops here.
>>> *
>>> **
>>> * * *
>>> In the same email, I invited you to get your hands dirty, and you have
>>> followed that invitation and you even got a feature in. That’s the right
>>> start, I welcome this. And now you are getting some feedback about pursuing
>>> this road and you don’t like it, and then you start with the derail again.
>>> Dynamic rewrites are feature with very limited use in the grander scope of
>>> things, so I have no qualms about having this in CouchDB.
>>> But as I outlined on numerous occasions now, the technical CouchApp
>>> architecture within CouchDB is a _dead-end_. It has not been touched for 5+
>>> years until the rewrites patch came in, it has serious design flaws and
>>> abysmal performance characteristics. It is literally a cgi-bin, which we
>>> collectively agreed at the end of the 90s is a terrible idea to serve
>>> dynamic content from.
>>> I’d welcome an app-server on top of CouchDB like you all want, but it
>>> won’t get any meaningful adoption of users or support from other devs if
>>> you build it on couchjs.
>>> I’ve long been in favour of porting our view server to Node.js. And if
>>> then every CouchDB also has a Node.js, you can build a *fantastic*, modern,
>>> app server platform on top of *that*. I wouldn’t even mind shipping this
>>> with CouchDB.
>>> But this won’t be CouchDB’s main story: A Database that replicates. A
>>> Database that has sibling-projects it replicates with to enable even more
>>> use-cases. A Database, that defines an open sync standard that powers the
>>> next 20 years in mobile/cluster computing. That’s where the power of this
>>> project lies, not in another app server of which there are plenty and of
>>> which many already work very well with CouchDB.
>>> FWIW, I’d work full-time on porting our view server to Node.js if I didn’t
>>> realise that shipping 2.0 and the work on the sync protocol are _way_ more
>>> important. So that’s what I’m working on. And that’s what everyone else
>>> here is working on. We need to come together as a team get this out, or 2.0
>>> will never ship.
>>> tl;dr:
>>> - we have limited resources and we have to pick our battles, and what we
>>> need to be doing is on a clear roadmap for the foreseeable future, and an
>>> app server is not it.
>>> - you won’t change the main mission statement of CouchDB until after we
>>> shipped 2.0 and 3.0 and whichever other release we need to get some
>>> significant revisions to the sync protocol into the user-base.
>>> - if you want to build a modern app server into CouchDB, do it with a
>>> better technical foundation. The road, at least to me, is obvious, it’s
>>> just a lot of work, like every fun endeavour.
>>> - if you want to do this concurrently with the other efforts going on,
>>> we’d be more than happy to establish a sub-project for you, where you can
>>> reign free, and not derail the efforts happening here, that the majority of
>>> the people doing the work here deem more important.
>>> * * *
>>> One final note that I just copy-and-paste out of the aforementioned
>>> thread: before you consider turning this into an ad-hominem attack, or some
>>> insinuation that I am abusing my PMC Chair position to push through my
>>> personal agenda or vendetta against you and your loved ones, or any of this
>>> sort of crap (that has come up before), keep it to yourself, thanks.
>>> Best
>>> Jan
>>> --
>>>> On 20 Oct 2015, at 11:10, Dale Harvey <> wrote:
>>>> This discussion has gone round and round a couple of times in different
>>>> forms
>>>> so I will avoid repeating my previous points but from working on PouchDB,
>>>> the focus on having 'PouchDB' be a database only is fairly liberating, by
>>>> not trying
>>>> to add fairly arbitrary "application platform" features into the core
>>>> codebase we can focus
>>>> on integrating with what does provide those application platform features
>>>> much better.
>>>> Users do not need to wait for reverse proxying or url rewriting to be an
>>>> agreed, implemented
>>>> and maintained feature, they can use the many that already exist and we
>>>> will ensure
>>>> that we integrate with their use case as best as we possibly can.
>>>> On 19 October 2015 at 23:57, Harald Kisch <> wrote:
>>>>> Hi Garren,
>>>>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying to
>>>>> support their users with markup languages like XML, HTML, etc. for
>>> instance
>>>>> directly out of the database core (performance, simplicity,
>>>>> scalability,..).
>>>>> Lotus Notes did also integrate JavaScript inside of their core (Do you
>>> know
>>>>> which guy did take part of it?). This have different reasons, but one
>>>>> this reasons is to support users with dynamic mutable data directly into
>>>>> their GUI in JSON format which in my opinion is the fundamental part
>>>>> CouchDB to be a database for the web.
>>>>> Improvements get lost if we look at others and try not to be different.
>>> In
>>>>> my opinion we should more think about replacing spidermonkey with the
>>>>> google V8 engine and itegrate node completely into the CouchDB core to
>>>>> consume npm-packages directly instead of using them in the local
>>> filesystem
>>>>> outside of CouchDB, where unfortunatelly complexity rise up at scaling.
>>>>> --Harald
>>>>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby <> wrote:
>>>>>> Hi Garren,
>>>>>> thanks for the "not standing in the way", I hope for more volunteers
>>>>>> iron out some of CouchDB's old akward wrinkles.
>>>>>> I am all with you for simplification:) ermouth's rewrite function
is a
>>>>>> huge simplifier.
>>>>>> Where I disagree with you is where you say "probably a sign that
>>>>> idea
>>>>>> is not something worth pursuing".
>>>>>> Whenever you discover that you have a differentiator, it's always
>>> good
>>>>>> idea to look closely before discaring it and blend in with the rest.
>>>>>> It's all about attracting the next million web developers.
>>>>>> johs
>>>>>>> On 19. okt. 2015, at 20.08, Garren Smith <>
>>>>>>> I’m really struggling with these proposals. I love the enthusiasm
>>>>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>>>>> CouchDB is ultimately a database. One with excellent sync
>>> capabilities.
>>>>>> And combining that with libraries like PouchDB and Hoodie make it
>>>>>> amazing database to build applications with.
>>>>>>> Adding routers and reverse proxies just makes it feel like we
>>> to
>>>>>> push CouchDB into being more than it needs to be.
>>>>>>> For example building Couchapp like functionality in Node.js is
>>>>> simple
>>>>>> and way better. Languages like Go also do that really well. Far
>>> superior
>>>>>> than what we can do with a database.
>>>>>>> I would rather let the Node.js and Go web libraries serve content
>>>>>> let us focus on building a clustered replicating database. We will
>>>>>> more people to this community if we can do that properly over creaky,
>>>>> slow
>>>>>> and limited web serving mashed on top of a database.
>>>>>>> If I look at other popular databases, I don’t see any of them
>>>>>> web content which is probably a sign that this idea is not something
>>>>> worth
>>>>>> pursuing.
>>>>>>> However if there is a burning desire for this and developers
>>>>>> their hands to code this functionality, I would not stand in your
>>> It
>>>>>> is great to see the varied use of CouchDB out in the wild.
>>>>>>> Cheers
>>>>>>> Garren
>>>>>>>> On 19 Oct 2015, at 4:47 PM, Johs. E <>
>>>>>>>> Thanks Andy,
>>>>>>>> I will try and  get some use cases up on confluence.
>>>>>>>> As for whoever would pick up the work after ermouth,  I have
>>> course
>>>>>> one big thing on the wish list that goes well with a new router
>>>>> solution..
>>>>>>>> reverse proxy
>>>>>>>> I remember asking about it when I first started to work w
CouchDB and
>>>>>> there were some concerns regarding security.
>>>>>>>> Since then I think node.js has paved the way with content
>>> and
>>>>>> all sorts of outgoing traffic.
>>>>>>>> Has anyone work on a reverse proxy solution for Couch?
>>>>>>>> johs
>>>>>>>>> On 18 Oct 2015, at 21:36, Andy Wenk <>
>>>>>>>>> Hey Johs,
>>>>>>>>> thanks a lot for this. I need some time to dig into it.
We need a
>>>>>> place to
>>>>>>>>> write the user stories / use case down. So I suggest
we find good
>>>>>> place at
>>>>>>>>> the cwiki. So I suggest to use
>>>>>> .
>>>>>>>>> Do you have write access there? If not, please ping me.
>>>>>>>>> Great work!
>>>>>>>>> All the best
>>>>>>>>> Andy
>>>>>>>>> P.S.: Jan already mentioned the feature freeze. Please
take it not
>>>>> as a
>>>>>>>>> demotivation but as the possibility to have a bit more
time to work
>>>>> on
>>>>>> it.
>>>>>>>>> On 17 October 2015 at 17:32, Johs Ensby <>
>>>>>>>>>> Andy,
>>>>>>>>>> I will make my first use case for function in _rewrite
a high level
>>>>>> one:
>>>>>>>>>> to create a standalone server that is an all-in-one
DB server,
>>>>>> application
>>>>>>>>>> server, api server and web server.
>>>>>>>>>> I have played with the build of CouchDB 2 with rewrite
>>>>>>>>>> implemented that  ermouth put up on the irish AWS
community AMI
>>> list
>>>>>> and
>>>>>>>>>> the new use cases are endless.
>>>>>>>>>> First, I find that there are a few things that people
fail to
>>> notice
>>>>>> about
>>>>>>>>>> ddocs.
>>>>>>>>>> you need a tool to build a ddoc, editing JSON is
not a viable
>>>>> option.
>>>>>> The
>>>>>>>>>> Ddoc Lab of ermouth is in a class of its own. If
you havent tried
>>> it
>>>>>> yet,
>>>>>>>>>> do so from <>.
Installing on your
>>>>> own
>>>>>>>>>> couch it is as easy as storing the application, all
included as one
>>>>>>>>>> document in any database. Ddoc Lab is a component
oriented IDE with
>>>>>> syntax
>>>>>>>>>> checking, less preprosessor and other build tools
that let you keep
>>>>> a
>>>>>> well
>>>>>>>>>> organized ddoc as a source project (in one couchdb
document) and
>>> you
>>>>>>>>>> publich a ddoc to any target db.
>>>>>>>>>> with this tool you can organize your js modules and
templates etc
>>>>> and
>>>>>>>>>> basically...
>>>>>>>>>> set up the API of your application in a ddoc. You
can switch
>>> between
>>>>>>>>>> databases and their ddoc functionality based on username,
role or
>>>>>>>>>> geolocation and limit access to parts of the Couch
API as needed
>>>>>>>>>> This is the method I would recommend to explore powerful
>>>>>> with
>>>>>>>>>> function in rewrites
>>>>>>>>>> redirect port 80 directly to couch
>>>>>>>>>> set up 2 vhosts, one for public access pointing to
>>> youdb/_design/api
>>>>>> and
>>>>>>>>>> one for sysadm pointing to /
>>>>>>>>>> for admin use Fauxton and Ddoc Lab on the sysadm
>>>>>>>>>> you are set to develop great systems, no big tool
stack to learn,
>>>>> just
>>>>>>>>>> bring in whatever js modules you like, the template
engine you
>>> like,
>>>>>> the
>>>>>>>>>> router you like, the HTML5 stuff you like..
>>>>>>>>>> .. or just write some very compact js code in one
place where you
>>>>>> ealier
>>>>>>>>>> had to mess around with a whole stack of tools and
>>>>>>>>>> below is the req object that the function takes
>>>>>>>>>> Johs
>>>>>>>>>> The rewrite function has this syntax
>>>>>>>>>> function(req) {
>>>>>>>>>>    .. your code that will
>>>>>>>>>>    return
>>>>>>>>>>            path
>>>>>>>>>>            // optional
>>>>>>>>>>            headers
>>>>>>>>>>            method // you can change this on the fly
>>>>>>>>>>            code
>>>>>>>>>>            body
>>>>>>>>>> }
>>>>>>>>>> the function receives this req object
>>>>>>>>>> method
>>>>>>>>>> path
>>>>>>>>>> raw_path
>>>>>>>>>> query
>>>>>>>>>> headers
>>>>>>>>>>    Accept
>>>>>>>>>>    Accept-Encoding
>>>>>>>>>>    Connection
>>>>>>>>>>    Host
>>>>>>>>>>    Upgrade-Insecure-Requests
>>>>>>>>>>    User-Agent
>>>>>>>>>>    x-couchdb-vhost-path
>>>>>>>>>> body
>>>>>>>>>> peer
>>>>>>>>>> cookie
>>>>>>>>>> userCtx
>>>>>>>>>> db
>>>>>>>>>> name
>>>>>>>>>> roles
>>>>>>>>>> secObj
>>>>>>>>>>> On 1. okt. 2015, at 13.40, Andy Wenk <>
>>>>>>>>>>> Johs,
>>>>>>>>>>> Yes for sure! That's always great. Maybe you
can also write some
>>>>> user
>>>>>>>>>> stories (given when then) or scribble some graphics.
Everything is
>>>>>> useful
>>>>>>>>>> and will fasten the process ;-)
>>>>>>>>>>> Cheers
>>>>>>>>>>> Andy
>>>>>>>>>>> On 1 Oct 2015 12:38, "Johs Ensby" <
>>>>>>>>>> wrote:
>>>>>>>>>>> Thanks for this Andy,
>>>>>>>>>>> I can contribute to the discussion of the feature
seen from a user
>>>>>>>>>> perspective.
>>>>>>>>>>> Would it be appropriate to present some use cases?
>>>>>>>>>>> best
>>>>>>>>>>> Johs
>>>>>>>>>>>> On 1. okt. 2015, at 12.33, Andy Wenk <
>>>>> <mailto:
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Johs,
>>>>>>>>>>>> Let me please show the steps needed.
>>>>>>>>>>>> * discuss the feature very clearly on the
dev@. Please make sure
>>>>>> that
>>>>>>>>>> core
>>>>>>>>>>>> developers as committers with commit bits
are involved
>>>>>>>>>>>> * code the feature. Make sure to implement
>>>>>>>>>>>> * send a pull request and show it to dev@
>>>>>>>>>>>> * finally the community will accept or decline
the feature (this
>>>>>> will
>>>>>>>>>>>> involve refactoring and changes)
>>>>>>>>>>>> As Alex said. The PMC or Jan do not decide
about the feature.
>>>>>>>>>>>> All the best
>>>>>>>>>>>> Andy
>>>>>>>>>>>> On 1 Oct 2015 11:21, "Alexander Shorin" <
>>>>> <mailto:
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On Thu, Oct 1, 2015 at 12:07 PM, Johs
Ensby <
>>>>> <mailto:
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> will you welcome ermouths rewrite
>>>>>>>>>>>>> The decision is depends on the implementation.
If it will be
>>>>> good,
>>>>>> why
>>>>>>>>>>>>> not? Finally, CouchDB is open source
project: we cannot forbid
>>>>>> people
>>>>>>>>>>>>> right for contributions, we only welcome
>>>>>>>>>>>>>> Arguments against couchapps has to
do with performance and the
>>>>>> folly
>>>>>>>>>> in
>>>>>>>>>>>>> competing with node.js.
>>>>>>>>>>>>> Performance question for the new _rewrite
implementation is very
>>>>>>>>>>>>> depends on query server. Once it can
process this kind of
>>>>>> functions,
>>>>>>>>>>>>> you may use something better than JS
to gain better performance.
>>>>>> That
>>>>>>>>>>>>> could be Erlang native query server,
or luerl-based one, or else
>>>>>> you
>>>>>>>>>>>>> like.
>>>>>>>>>>>>> --
>>>>>>>>>>>>> ,,,^..^,,,
>>>>>>>>> --
>>>>>>>>> Andy Wenk
>>>>>>>>> Hamburg - Germany
>>>>>>>>> RockIt!
>>>>>>>>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3
9ED3 9588
>>> --
>>> Professional Support for Apache CouchDB:

Professional Support for Apache CouchDB:

View raw message