couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: Erlang vs JavaScript
Date Thu, 15 Aug 2013 09:57:48 GMT

On Aug 15, 2013, at 11:53 , Benoit Chesneau <bchesneau@gmail.com> wrote:

> On Thu, Aug 15, 2013 at 11:38 AM, Jan Lehnardt <jan@apache.org> wrote:
> 
>> 
>> On Aug 15, 2013, at 10:09 , Robert Newson <rnewson@apache.org> wrote:
>> 
>>> A big +1 to Jason's clarification of "erlang" vs "native". CouchDB
>>> could have shipped an erlang view server that worked in a separate
>>> process and had the stdio overhead, to combine the slowness of the
>>> protocol with the obtuseness of erlang. ;)
>>> 
>>> Evaluating Javascript within the erlang VM process intrigues me, Jens,
>>> how is that done in your case? I've not previously found the assertion
>>> that V8 would be faster than SpiderMonkey for a view server compelling
>>> since the bottleneck is almost never in the code evaluation, but I do
>>> support CouchDB switching to it for the synergy effects of a closer
>>> binding with node.js, but if it's running in the same process, that
>>> would change (though I don't immediately see why the same couldn't be
>>> done for SpiderMonkey). Off the top of my head, I don't know a safe
>>> way to evaluate JS in the VM. A NIF-based approach would either be
>>> quite elaborate or would trip all the scheduling problems that
>>> long-running NIF's are now notorious for.
>>> 
>>> At a step removed, the view server protocol itself seems like the
>>> thing to improve on, it feels like that's the principal bottleneck.
>> 
>> The code is here:
>> https://github.com/couchbase/couchdb/tree/master/src/mapreduce
>> 
>> I’d love for someone to pick this up and give CouchDB, say, a ./configure
>> --enable-native-v8 option or a plugin that allows people to opt into the
>> speed improvements made there. :)
>> 
>> The choice for V8 was made because of easier integration API and more
>> reliable releases as a standalone project, which I think was a smart move.
>> 
>> IIRC it relies on a change to CouchDB-y internals that has not made it
>> back from Couchbase to CouchDB (Filipe will know, but I doubt he’s reading
>> this thread), but we should look into that and get us “native JS views”, at
>> least as an option or plugin.
>> 
>> CCing dev@.
>> 
>> Jan
>> --
>> 
>> 
> Well on the first hand nifs look like a good idea but can be very
> problematic:
> 
> - when the view computation take time it would block the full vm
> scheduling. It can be mitigated using a pool of threads to execute the work
> asynchronously but then can create other problems like memory leaking etc.
> - nifs can't be upgraded easily during hot upgrade
> - when a nif crash, all the vm crash.

Yeah totally, hence making the whole thing an option.

> (Note that we have the same problem when using a nif to decode/encode json,
> it only works well with medium sized documents)




> One other way to improve the js handling would be removing the main
> bottleneck ie the serialization-deserialization we do on each step. Not
> sure if it exists but  feasible, why not passing erlang terms from erlang
> to js and js to erlang? So at the end the deserialization would happen only
> on the JS side ie instead of having
> 
> get erlang term
> encode to json
> send to js
> decode json
> process
> encode json
> send json
> decode json to erlang term
> store
> 
> we sould just have
> 
> get erlang term
> send over STDIO
> decode erlang term to JS object
> process
> encode to erlang term
> send erlang term
> store
> 
> Erlang serialization is also very optimised.
> 
> 
> Both solutions could co-exist, that may worh a try and benchmark each...

I think we just want both solutions period, the embedded one will still be faster, but potentially
a little less stable, and the external view server one will be slower but extremely robust.
Users should be able to choose between them :)

Best
Jan
--



Mime
View raw message