couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <>
Subject Re: [PROPOSAL] Allow rewrites to be JS function
Date Wed, 21 Oct 2015 11:49:16 GMT

> On 21 Oct 2015, at 13:08, Harald Kisch <> wrote:
>> This project doesn’t have the capacity to get a distributed syncing
> database AND an application server platform going > at the same time.
> So please explain: Why CouchAppers get bothered every time when CouchApps
> comes to discussion?
> I thought the CouchApp discussions are very welcome @dev. Is'nt it the fact
> that this mailing list is exactly for this kind of discussions? Is'nt it
> the fact that this negative emails always say: "Do not work on this."
> Something in me is crying when I read something like that because until
> now, I did'nt read any hard fact WHY NOT?
> Here the dog bites in his own tail: "Why couchappers are not allowed to
> work on CouchApps in between the CouchDB Community anymore and at the same
> time the argument rises that it was not touched since 5 years”?

You are twisting my words. I said I welcome the recent work. What I don’t
welcome is trying to convince dev@ that CouchApp is somehow the future and
big chance of CouchDB when we already have decided that that’s not the case,
in the light of the least resistance about your contribution.

> As far as I know, there was no request and no claim made for extending
> CouchApp features,

You haven’t been looking very thoroughly then.

> but I feel compromized and confronted with very bad energy when it comes
> to this topic.

Yes, because we have wasted a lot of developer time and end-user goodwill
by getting sidetracked with CouchApps (I’m principally guilty of that, btw.),
and we have since realised that we have to double and triple down on the
database core as outlined instead of getting entangled in CouchApps again.

To the people who argue against you, CouchApps were, in hindsight, a very
painful experience that ultimately held back the potential of CouchDB, and
we are not prepared to do that mistake again.

> I think there are many developers who
> would love to start coding on CouchApps but not under the hood of current
> circumstances with getting bothered every time a mail appears @marketing or
> @dev concerning CouchApps.

Hence the proposal to fork this off into its own safe-space where we can
focus on the core database and CouchApp* people can advance the technology.

> I do not understand where this negative emotions against CouchApps comes
> from. As mentioned before: There are people doing business on top of
> CouchApps. It can't be buggy, incomplete, insecure or something else when
> people start to have business on top of it.

You mean like Microsoft and Apple ship bug-free, complete and secure
software? #scnr

I outlined the reasons why CouchApps are not very welcome here above and
in previous emails: we don’t have the capacity to do this as a project at
this point in time and the technical underpinnings need a complete re-work.

> Instead of listening to them,
> which solutions they have found, they get botherd. Could you explain
> please, what is the reason for this emotional emails?

Like I said, this all has been a painful evolution five, six years back and
I don’t fault anyone new to the project for not understanding this. But from
what you’ve heard from the people who’ve been running the place here, you
obviously have gotten some resistance and it has been explained to you a number
of times and now it feels to us like trying to deal with a child throwing a
temper-tantrum because it’s not getting what they want. That’s utterly frustrating
in a professional environment.

In addition, as you personally very well know, we are dealing with a lot more
than is just visible on dev@ and when we say “we don’t have the time and 
resources to deal with this right now”, and when then the topic is brought up
again and again, all we can conclude is that somebody is interested in actively
harming the CouchDB project, and we’ll react accordingly.

> If there is a real
> hard fact not using CouchApps, I promise, I will stop writing emails
> regarding CouchApps immediatly for any CouchDB mailing list.

I suggested many roads out of this pickle:

1. create a space where progress of CouchApps can be discussed
2. work on a technically sound solution to a next generation CouchApp tech
   stack (I even gave a pretty good hint on how this can play out).
3. ship

There is no need to throw logical fallacies at us (of course we can’t prove there
are no CouchApp users, and you know that) to coax us into a discussion we
have had many times and are ready to not have again for the foreseeable future.

Open source means everyone can contribute, but it doesn’t mean your contribution
necessarily makes it into the project.

If there are viable business built on CouchApps, the resources to get this going
should come together quickly.


> --Harald
> I would say, many people lost their interest in CouchDB because exactly for
> such claims like:
>> *
>> **
> On Tue, Oct 20, 2015 at 9:09 PM, Jan Lehnardt <> wrote:
>> What Dale and the others said.
>> And what I explained at length on the marketing@ mailing list to the very
>> same people:
>> This project doesn’t have the capacity to get a distributed syncing
>> database AND an application server platform going at the same time. And the
>> people who are doing the lion’s share of the work here have long concluded
>> that we need to get the database part going before we can tackle anything
>> else.
>> Like others said, CouchDB is flexible enough to let anyone build this on
>> top of, or make a plugin, or whatever, we can even make it an ASF
>> sub-project, but this needs to happen separately from what we’ve all agreed
>> in 2012 and again in 2014 and 15 to do here, which is ship a kick-ass
>> database, that replicates.
>> The one “extra-curricular” thing we have to allow ourselves is pushing the
>> sync protocol forward, together with PouchDB & friends, and accommodate the
>> new use cases that our syncing cousins enable for our end-users.
>> Things like selective sync, a less chatty sync protocol for mobile users
>> (_bulk_get/_revs is only a small patch for a larger problem). These are the
>> pressing issues for this project that we have to get right before we can
>> consider building an app server on top of it. And we have to ship the 2.0
>> cluster first, too, so anyone with JavaScript chops here should join the
>> “The JavaScript Test Suite” thread and get hacking.
>> If you are interested in building an app server on top of CouchDB, do
>> that: build it on top of CouchDB. We are more than happy put you into the
>> Apache family and help with naming, community, security, distribution etc.
>> issues, but this won’t be the core focus of this project.
>> And like I mentioned to you very same people on marketing@ before*,**:
>> I’m tired of the deliberate derailing of this project.
>> It stops here.
>> *
>> **
>> * * *
>> In the same email, I invited you to get your hands dirty, and you have
>> followed that invitation and you even got a feature in. That’s the right
>> start, I welcome this. And now you are getting some feedback about pursuing
>> this road and you don’t like it, and then you start with the derail again.
>> Dynamic rewrites are feature with very limited use in the grander scope of
>> things, so I have no qualms about having this in CouchDB.
>> But as I outlined on numerous occasions now, the technical CouchApp
>> architecture within CouchDB is a _dead-end_. It has not been touched for 5+
>> years until the rewrites patch came in, it has serious design flaws and
>> abysmal performance characteristics. It is literally a cgi-bin, which we
>> collectively agreed at the end of the 90s is a terrible idea to serve
>> dynamic content from.
>> I’d welcome an app-server on top of CouchDB like you all want, but it
>> won’t get any meaningful adoption of users or support from other devs if
>> you build it on couchjs.
>> I’ve long been in favour of porting our view server to Node.js. And if
>> then every CouchDB also has a Node.js, you can build a *fantastic*, modern,
>> app server platform on top of *that*. I wouldn’t even mind shipping this
>> with CouchDB.
>> But this won’t be CouchDB’s main story: A Database that replicates. A
>> Database that has sibling-projects it replicates with to enable even more
>> use-cases. A Database, that defines an open sync standard that powers the
>> next 20 years in mobile/cluster computing. That’s where the power of this
>> project lies, not in another app server of which there are plenty and of
>> which many already work very well with CouchDB.
>> FWIW, I’d work full-time on porting our view server to Node.js if I didn’t
>> realise that shipping 2.0 and the work on the sync protocol are _way_ more
>> important. So that’s what I’m working on. And that’s what everyone else
>> here is working on. We need to come together as a team get this out, or 2.0
>> will never ship.
>> tl;dr:
>> - we have limited resources and we have to pick our battles, and what we
>> need to be doing is on a clear roadmap for the foreseeable future, and an
>> app server is not it.
>> - you won’t change the main mission statement of CouchDB until after we
>> shipped 2.0 and 3.0 and whichever other release we need to get some
>> significant revisions to the sync protocol into the user-base.
>> - if you want to build a modern app server into CouchDB, do it with a
>> better technical foundation. The road, at least to me, is obvious, it’s
>> just a lot of work, like every fun endeavour.
>> - if you want to do this concurrently with the other efforts going on,
>> we’d be more than happy to establish a sub-project for you, where you can
>> reign free, and not derail the efforts happening here, that the majority of
>> the people doing the work here deem more important.
>> * * *
>> One final note that I just copy-and-paste out of the aforementioned
>> thread: before you consider turning this into an ad-hominem attack, or some
>> insinuation that I am abusing my PMC Chair position to push through my
>> personal agenda or vendetta against you and your loved ones, or any of this
>> sort of crap (that has come up before), keep it to yourself, thanks.
>> Best
>> Jan
>> --
>>> On 20 Oct 2015, at 11:10, Dale Harvey <> wrote:
>>> This discussion has gone round and round a couple of times in different
>>> forms
>>> so I will avoid repeating my previous points but from working on PouchDB,
>>> the focus on having 'PouchDB' be a database only is fairly liberating, by
>>> not trying
>>> to add fairly arbitrary "application platform" features into the core
>>> codebase we can focus
>>> on integrating with what does provide those application platform features
>>> much better.
>>> Users do not need to wait for reverse proxying or url rewriting to be an
>>> agreed, implemented
>>> and maintained feature, they can use the many that already exist and we
>>> will ensure
>>> that we integrate with their use case as best as we possibly can.
>>> On 19 October 2015 at 23:57, Harald Kisch <> wrote:
>>>> Hi Garren,
>>>> look at MSSQL (CLR) or ORACLE (JAVA Forms) any database was trying to
>>>> support their users with markup languages like XML, HTML, etc. for
>> instance
>>>> directly out of the database core (performance, simplicity,
>>>> scalability,..).
>>>> Lotus Notes did also integrate JavaScript inside of their core (Do you
>> know
>>>> which guy did take part of it?). This have different reasons, but one of
>>>> this reasons is to support users with dynamic mutable data directly into
>>>> their GUI in JSON format which in my opinion is the fundamental part of
>>>> CouchDB to be a database for the web.
>>>> Improvements get lost if we look at others and try not to be different.
>> In
>>>> my opinion we should more think about replacing spidermonkey with the
>>>> google V8 engine and itegrate node completely into the CouchDB core to
>>>> consume npm-packages directly instead of using them in the local
>> filesystem
>>>> outside of CouchDB, where unfortunatelly complexity rise up at scaling.
>>>> --Harald
>>>> On Mon, Oct 19, 2015 at 9:24 PM, Johs Ensby <> wrote:
>>>>> Hi Garren,
>>>>> thanks for the "not standing in the way", I hope for more volunteers
>>>>> iron out some of CouchDB's old akward wrinkles.
>>>>> I am all with you for simplification:) ermouth's rewrite function is
>>>>> huge simplifier.
>>>>> Where I disagree with you is where you say "probably a sign that this
>>>> idea
>>>>> is not something worth pursuing".
>>>>> Whenever you discover that you have a differentiator, it's always a
>> good
>>>>> idea to look closely before discaring it and blend in with the rest.
>>>>> It's all about attracting the next million web developers.
>>>>> johs
>>>>>> On 19. okt. 2015, at 20.08, Garren Smith <>
>>>>>> I’m really struggling with these proposals. I love the enthusiasm
>>>>> everyone but I keep thinking we should rather simplify CouchDB.
>>>>>> CouchDB is ultimately a database. One with excellent sync
>> capabilities.
>>>>> And combining that with libraries like PouchDB and Hoodie make it an
>>>>> amazing database to build applications with.
>>>>>> Adding routers and reverse proxies just makes it feel like we trying
>> to
>>>>> push CouchDB into being more than it needs to be.
>>>>>> For example building Couchapp like functionality in Node.js is so
>>>> simple
>>>>> and way better. Languages like Go also do that really well. Far
>> superior
>>>>> than what we can do with a database.
>>>>>> I would rather let the Node.js and Go web libraries serve content
>>>>> let us focus on building a clustered replicating database. We will draw
>>>>> more people to this community if we can do that properly over creaky,
>>>> slow
>>>>> and limited web serving mashed on top of a database.
>>>>>> If I look at other popular databases, I don’t see any of them serving
>>>>> web content which is probably a sign that this idea is not something
>>>> worth
>>>>> pursuing.
>>>>>> However if there is a burning desire for this and developers raising
>>>>> their hands to code this functionality, I would not stand in your way.
>> It
>>>>> is great to see the varied use of CouchDB out in the wild.
>>>>>> Cheers
>>>>>> Garren
>>>>>>> On 19 Oct 2015, at 4:47 PM, Johs. E <> wrote:
>>>>>>> Thanks Andy,
>>>>>>> I will try and  get some use cases up on confluence.
>>>>>>> As for whoever would pick up the work after ermouth,  I have
>> course
>>>>> one big thing on the wish list that goes well with a new router
>>>> solution..
>>>>>>> reverse proxy
>>>>>>> I remember asking about it when I first started to work w CouchDB
>>>>> there were some concerns regarding security.
>>>>>>> Since then I think node.js has paved the way with content scraping
>> and
>>>>> all sorts of outgoing traffic.
>>>>>>> Has anyone work on a reverse proxy solution for Couch?
>>>>>>> johs
>>>>>>>> On 18 Oct 2015, at 21:36, Andy Wenk <>
>>>>>>>> Hey Johs,
>>>>>>>> thanks a lot for this. I need some time to dig into it. We
need a
>>>>> place to
>>>>>>>> write the user stories / use case down. So I suggest we find
>>>>> place at
>>>>>>>> the cwiki. So I suggest to use
>>>>> .
>>>>>>>> Do you have write access there? If not, please ping me.
>>>>>>>> Great work!
>>>>>>>> All the best
>>>>>>>> Andy
>>>>>>>> P.S.: Jan already mentioned the feature freeze. Please take
it not
>>>> as a
>>>>>>>> demotivation but as the possibility to have a bit more time
to work
>>>> on
>>>>> it.
>>>>>>>> On 17 October 2015 at 17:32, Johs Ensby <>
>>>>>>>>> Andy,
>>>>>>>>> I will make my first use case for function in _rewrite
a high level
>>>>> one:
>>>>>>>>> to create a standalone server that is an all-in-one DB
>>>>> application
>>>>>>>>> server, api server and web server.
>>>>>>>>> I have played with the build of CouchDB 2 with rewrite
>>>>>>>>> implemented that  ermouth put up on the irish AWS community
>> list
>>>>> and
>>>>>>>>> the new use cases are endless.
>>>>>>>>> First, I find that there are a few things that people
fail to
>> notice
>>>>> about
>>>>>>>>> ddocs.
>>>>>>>>> you need a tool to build a ddoc, editing JSON is not
a viable
>>>> option.
>>>>> The
>>>>>>>>> Ddoc Lab of ermouth is in a class of its own. If you
havent tried
>> it
>>>>> yet,
>>>>>>>>> do so from <>. Installing
on your
>>>> own
>>>>>>>>> couch it is as easy as storing the application, all included
as one
>>>>>>>>> document in any database. Ddoc Lab is a component oriented
IDE with
>>>>> syntax
>>>>>>>>> checking, less preprosessor and other build tools that
let you keep
>>>> a
>>>>> well
>>>>>>>>> organized ddoc as a source project (in one couchdb document)
>> you
>>>>>>>>> publich a ddoc to any target db.
>>>>>>>>> with this tool you can organize your js modules and templates
>>>> and
>>>>>>>>> basically...
>>>>>>>>> set up the API of your application in a ddoc. You can
>> between
>>>>>>>>> databases and their ddoc functionality based on username,
role or
>>>>>>>>> geolocation and limit access to parts of the Couch API
as needed
>>>>>>>>> This is the method I would recommend to explore powerful
>>>>> with
>>>>>>>>> function in rewrites
>>>>>>>>> redirect port 80 directly to couch
>>>>>>>>> set up 2 vhosts, one for public access pointing to
>> youdb/_design/api
>>>>> and
>>>>>>>>> one for sysadm pointing to /
>>>>>>>>> for admin use Fauxton and Ddoc Lab on the sysadm vhost
>>>>>>>>> you are set to develop great systems, no big tool stack
to learn,
>>>> just
>>>>>>>>> bring in whatever js modules you like, the template engine
>> like,
>>>>> the
>>>>>>>>> router you like, the HTML5 stuff you like..
>>>>>>>>> .. or just write some very compact js code in one place
where you
>>>>> ealier
>>>>>>>>> had to mess around with a whole stack of tools and systems
>>>>>>>>> below is the req object that the function takes
>>>>>>>>> Johs
>>>>>>>>> The rewrite function has this syntax
>>>>>>>>> function(req) {
>>>>>>>>>    .. your code that will
>>>>>>>>>    return
>>>>>>>>>            path
>>>>>>>>>            // optional
>>>>>>>>>            headers
>>>>>>>>>            method // you can change this on the fly
>>>>>>>>>            code
>>>>>>>>>            body
>>>>>>>>> }
>>>>>>>>> the function receives this req object
>>>>>>>>> method
>>>>>>>>> path
>>>>>>>>> raw_path
>>>>>>>>> query
>>>>>>>>> headers
>>>>>>>>>    Accept
>>>>>>>>>    Accept-Encoding
>>>>>>>>>    Connection
>>>>>>>>>    Host
>>>>>>>>>    Upgrade-Insecure-Requests
>>>>>>>>>    User-Agent
>>>>>>>>>    x-couchdb-vhost-path
>>>>>>>>> body
>>>>>>>>> peer
>>>>>>>>> cookie
>>>>>>>>> userCtx
>>>>>>>>> db
>>>>>>>>> name
>>>>>>>>> roles
>>>>>>>>> secObj
>>>>>>>>>> On 1. okt. 2015, at 13.40, Andy Wenk <>
>>>>>>>>>> Johs,
>>>>>>>>>> Yes for sure! That's always great. Maybe you can
also write some
>>>> user
>>>>>>>>> stories (given when then) or scribble some graphics.
Everything is
>>>>> useful
>>>>>>>>> and will fasten the process ;-)
>>>>>>>>>> Cheers
>>>>>>>>>> Andy
>>>>>>>>>> On 1 Oct 2015 12:38, "Johs Ensby" <
>>>>>>>>> wrote:
>>>>>>>>>> Thanks for this Andy,
>>>>>>>>>> I can contribute to the discussion of the feature
seen from a user
>>>>>>>>> perspective.
>>>>>>>>>> Would it be appropriate to present some use cases?
>>>>>>>>>> best
>>>>>>>>>> Johs
>>>>>>>>>>> On 1. okt. 2015, at 12.33, Andy Wenk <
>>>> <mailto:
>>>>>>>>>>> wrote:
>>>>>>>>>>> Johs,
>>>>>>>>>>> Let me please show the steps needed.
>>>>>>>>>>> * discuss the feature very clearly on the dev@.
Please make sure
>>>>> that
>>>>>>>>> core
>>>>>>>>>>> developers as committers with commit bits are
>>>>>>>>>>> * code the feature. Make sure to implement tests
>>>>>>>>>>> * send a pull request and show it to dev@
>>>>>>>>>>> * finally the community will accept or decline
the feature (this
>>>>> will
>>>>>>>>>>> involve refactoring and changes)
>>>>>>>>>>> As Alex said. The PMC or Jan do not decide about
the feature.
>>>>>>>>>>> All the best
>>>>>>>>>>> Andy
>>>>>>>>>>> On 1 Oct 2015 11:21, "Alexander Shorin" <
>>>> <mailto:
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On Thu, Oct 1, 2015 at 12:07 PM, Johs Ensby
>>>> <mailto:
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> will you welcome ermouths rewrite contribution?
>>>>>>>>>>>> The decision is depends on the implementation.
If it will be
>>>> good,
>>>>> why
>>>>>>>>>>>> not? Finally, CouchDB is open source project:
we cannot forbid
>>>>> people
>>>>>>>>>>>> right for contributions, we only welcome
>>>>>>>>>>>>> Arguments against couchapps has to do
with performance and the
>>>>> folly
>>>>>>>>> in
>>>>>>>>>>>> competing with node.js.
>>>>>>>>>>>> Performance question for the new _rewrite
implementation is very
>>>>>>>>>>>> depends on query server. Once it can process
this kind of
>>>>> functions,
>>>>>>>>>>>> you may use something better than JS to gain
better performance.
>>>>> That
>>>>>>>>>>>> could be Erlang native query server, or luerl-based
one, or else
>>>>> you
>>>>>>>>>>>> like.
>>>>>>>>>>>> --
>>>>>>>>>>>> ,,,^..^,,,
>>>>>>>> --
>>>>>>>> Andy Wenk
>>>>>>>> Hamburg - Germany
>>>>>>>> RockIt!
>>>>>>>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3
9ED3 9588
>> --
>> Professional Support for Apache CouchDB:

Professional Support for Apache CouchDB:

View raw message