Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of mikeal.rogers@gmail.com
 designates 209.85.214.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=L/WhqMC0uJFbixBfo8JhCzveIwZeSFI6ewuuCywuW0Bt5PteMeEoGxQvLIN94khCQk
         ByJ0ceW+g+VUH3HQzObA4rrLxHaMHc3foLf0cYFMeYfc4g4azGblcl4U/+lP4+O9JtkL
         SNWT5V2zaxbWsWZYS3emrRQNHVDqBVWZkhxx8=
MIME-Version: 1.0
In-Reply-To: <AANLkTincb6PdwHtPYC7i2ZX25A+7qf2f7tvh3gEUytnr@mail.gmail.com>
References: <AANLkTin=PhtkbQX7SJSoDAgcs3EJ+VZ_BBsKb2fukwnr@mail.gmail.com>
	<AANLkTinte65oBBY1iDhjaV2e4QOrHAs60d0ZBqPb+Hst@mail.gmail.com>
	<AANLkTik1E-NzesS3hxO7zdtSdtM2eVscpcSULOud11Jj@mail.gmail.com>
	<AANLkTimfXM9YM1+TLRuF_SgD_LRhw+Z20vC9bmfg7ikv@mail.gmail.com>
	<AANLkTinF3Ypzk5ujTwD5mr=8v7sJcKkXbTjBAJ_GEDMC@mail.gmail.com>
	<AANLkTinbuVM+bK3=KgqcckgoJM9n1MjRfO3eUW0mmNWg@mail.gmail.com>
	<AANLkTinobdPn+k8fWFmS3OxDBQLALfn=x+9GkLaicsvP@mail.gmail.com>
	<AANLkTintbE3qZkK4c3Xz---fU35tnEh22r2Hib+9+MoJ@mail.gmail.com>
	<AANLkTi=kazQfTsYicZbiKTcqovy9d4Fg5=SswTdAW627@mail.gmail.com>
	<AANLkTinVafxTLYjvyEPRgRPgrwG1a2hzOuBRuo-xGOzM@mail.gmail.com>
	<AANLkTikZ0aYZim01gEFaQOeXgZYcxST074sa5yvF6D_q@mail.gmail.com>
	<AANLkTimDRfYfV3ZhOrD69VoWom0tCbr+c93zVwWkQ2yV@mail.gmail.com>
	<AANLkTimwq+0=CZG_YdJpX_=2V9kVQymkOhQBub7wYPet@mail.gmail.com>
	<AANLkTimujNeUyEvBKxvQnchUNf_Tgxmjc5c6Ew62AHQo@mail.gmail.com>
	<AANLkTincb6PdwHtPYC7i2ZX25A+7qf2f7tvh3gEUytnr@mail.gmail.com>
Date: Mon, 2 Aug 2010 14:59:40 -0700
Message-ID: <AANLkTi=cL14tP7XCjRKD5wQQ7Uk2t2rxzDE1e0DXu5NJ@mail.gmail.com>
Subject: Re: Proposal for changes in view server/protocol
From: Mikeal Rogers <mikeal.rogers@gmail.com>
To: dev@couchdb.apache.org
Content-Type: multipart/alternative; boundary=0016367d6be24186cd048cde512f

--0016367d6be24186cd048cde512f
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Aug 2, 2010 at 1:11 PM, Paul Davis <paul.joseph.davis@gmail.com>wrote:

> On Mon, Aug 2, 2010 at 2:49 PM, Mikeal Rogers <mikeal.rogers@gmail.com>
> wrote:
> > Back up a few steps.
> >
> > The view server line protocol while being terribly defined and taking
> > breaking changes in almost every release has still maintained a fair
> amount
> > of alternate implementations in languages other than JavaScript. I
> seriously
> > doubt that we would see this kind of diverse additions to CouchDB if the
> > barrier to integrating was writing erlang. We shouldn't fix fundamental
> > problems with the view server protocol and architecture solely by moving
> it
> > in to erlang.
>
> Even if we moved to a JS NIF runtime, we should still support the line
> protocol so that other languages can be used. Internally it would
> require some refactoring, but I think this is completely reasonable.
> And the refactoring probably needs to happen regardless.
>

This makes we far less worried. Thanks for clarifying Paul.


>
> > I'm also *highly* skeptical of the perceived performance benefits of
> moving
> > the js in to erlang compared to moving away from full JSON parsing and
> just
> > doing selective parsing/validation and storing the JSON string as opposed
> to
> > the native erlang representation. I can write this idea up in greater
> detail
> > in another thread but the native JSON parser in *any* language is likely
> to
> > be faster than something that parses the native erlang terms so it's only
> > faster on the erlang side to not hit the serialization again which we
> would
> > get if we stored the string+parsed underscore properties instead of the
> full
> > native erlang terms.
>
> I'm also *highly* skeptical that storing the string and selective
> parsing is going to do anything but complicate things even further. :D
>
> I'm not sure what you're comparing the native JSON parser too, but I
> can say that parsing Erlang terms is much easier and faster than JSON.
> Although, what's even *more* faster, is not using JSON at all. In the
> code that runs the NIF stuff I translate directly from Erlang terms to
> Spidermonkey objects programattically, so there's zero
> serialization/deserialization overhead. I haven't timed this code
> against couchjs yet, but I did a similar thing in passing Erlang terms
> to the view server and going from the external format to JavaScript
> objects directly which gave a noticeable speed up IIRC. It just
> happened to add about 1K lines of code that didn't seem like something
> I could require of other view implementations.
>

Definitely, passing Erlang terms will serialize and parse faster than JSON.
But selective JSON parsing on write and storing the string means that the
amount of times we currently do serialization whatsoever drops. Every time
we do a read or send something to the view server we just send that string,
pre-serialized strings are faster than JSON or erlang terms :)

Although this wasn't clear in my previous email, the selective JSON parsing
and js in NIF aren't mutually exclusive.


>
> > Not to mention the security and durability considerations of moving the
> view
> > server in to erlang. There is a reason erlang views aren't enable by
> > default.
> >
>
> The reason that Erlang isn't enabled by default is because its not
> sandbox-able. Or, more specifically, I couldn't figure out how to
> sandbox it. JS as a NIF is so sandboxed I still can't figure out how
> to make it communicate at all with Erlang without returning to the top
> of the stack.


> > -Mikeal
> >
> >
> > On Mon, Aug 2, 2010 at 11:39 AM, Jason Smith <jhs@couch.io> wrote:
> >
> >> On Tue, Aug 3, 2010 at 01:35, Randall Leeds <randall.leeds@gmail.com>
> >> wrote:
> >>
> >> > Either way we're waiting for a response from couch to call back into
> >> > JS, it's just whether we're calling the original function or the
> >> > continuation.
> >> > I don't have a strong preference either way, but I do slightly favor
> >> > the callback continuation as a mild style preference.
> >> >
> >>
> >> I have to say, callback style is way nicer. I have not thought about how
> >> that would look from non-Javascript query servers though.
> >>
> >>
> >> > In either case, do we have to add some new response format so the
> >> > query server can say "Hey call me again and tell me what to do with
> >> > this result?"
> >> > In other words, both of our suggestions can work on top of the same
> >> > change that needs to happen to the view server line protocol, so you
> >> > can code up either one and we can settle on what it should look like
> >> > to people writing update handlers later, no?
> >> >
> >>
> >> I suppose so. But I am unclear because earlier in the thread I posted
> >> working code that allows me to handle save conflicts (or validation
> >> failures) from within javascript, with no change to the line protocol.
> >> (That
> >> is about all it has going for it in fact!)
> >>
> >> --
> >> Jason Smith
> >> Couchio Hosting
> >>
> >
>

--0016367d6be24186cd048cde512f--