Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of bchesneau@gmail.com designates
 74.125.83.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAJ_m3YC9w0XDtn0zs=g35oZnT6hgmn3w=XPaoP3mtDdG1DgyfA@mail.gmail.com>
References: 
 <CAJNb-9rw_hpBznCg0Q3schhOwf9mEbH2h3AqhLFZXWussoFUdg@mail.gmail.com>
	<CAJ_m3YDGs0JWkv+0SVUJ80+nUGe51-C9-eR8wcinS_BaqOa63A@mail.gmail.com>
	<CAJNb-9o1_R5m-JQY6N3gw5mSX77Kis2OWQc8pTk2mu0AwRyZ=A@mail.gmail.com>
	<CAJ_m3YC9w0XDtn0zs=g35oZnT6hgmn3w=XPaoP3mtDdG1DgyfA@mail.gmail.com>
Date: Thu, 12 Jul 2012 09:53:42 +0200
Message-ID: 
 <CAJNb-9p3C1DowXOohPxDXBR4VEE-gQ+efsCxiROfch_q0PiOPQ@mail.gmail.com>
Subject: Re: couch_query_server refactoring
From: Benoit Chesneau <bchesneau@gmail.com>
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Jul 12, 2012 at 9:23 AM, Paul Davis <paul.joseph.davis@gmail.com> w=
rote:
> I actually see one of two ways. And which one we go with depends on
> and odd question that I've never really asked (or seen discussed)
> before.
>
> Both patterns would have a single couchjs process that is threaded and
> can respond to multiple requests simultaneously. Doing this gives us
> an ability to maximize throughput using pipelining and the like. The
> downfall is that if the couchjs process dies then it affects every
> other message that was in transit through it. Although I think we can
> mitigate a lot of this with some basic retry/exponential back off
> logic.
>
> One way we can do this is to use a similar approach to what we do now
> but asyncronously. Ie, messages from Erlang to couchjs become tagged
> tuples that a central process dispatches back and forth to clients.
> There are a few issues here. Firstly, we're going to basically need
> two caches of lookups which could be harmful to low latency
> requirements. Ie, Erlang will have to keep a cache of tag/client
> pairs, and the couchjs side will have to have some sort of lookup
> table/cache thing for ddoc ids to JS_Contexts. The weird question I
> have about this is, "What is the latency/bandwidth of stdio?" I've
> never thought to try and benchmark such a thing but it seems possible
> (no idea how likely though) that we could max out the capacity of a
> single file descriptor for communication.
>
> The second approach is to turn couchjs into a simple threaded network
> server. This way the Erlang side would just open a socket per design
> doc/context pair, and the couchjs side would be rather simple to
> implement (considering the alternative). I like this approach because
> it minimizes cache/lookups, uses more file descriptors (not sure if
> this is a valid concern), and (most importantly) keeps most of the
> complexity in Erlang where its easier to manage.


I quite like this second approach too. It would ease any
implementation and is also really portable. Ie passing a socket path
or port to an external server is easier in that case. For example on
IOS the app accessing to couchdb could maintain this process
internally and won't break any sandbox. It may be easier than writing
a nif for each cases...

>
> Either way, once either of those things is implemented we can
> re-implement our external process management to not totally bork
> servers under load as well as (hopefully) improve latency considerably
> for any of our "RPC" things.

you mean the pool of couch_os_processes ?

- beno=EEt