Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Date: Sun, 3 Apr 2011 20:14:05 +0000 (UTC)
From: "Adam Kocoloski (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Message-ID: 
 <1738921774.31446.1301861645782.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <545314120.30605.1301756765919.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (COUCHDB-1118) Adding a NIF based JSON
 decoding/encoding module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/COUCHDB-1118?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13=
015204#comment-13015204 ]=20

Adam Kocoloski commented on COUCHDB-1118:
-----------------------------------------

Hi Filipe, which version are you proposing to commit?

There must be some overhead associated with using a NIF.  I thought in at l=
east one version of Paul's work the overhead made the NIF version slower at=
 decoding very small documents than pure Erlang.  Is that no longer the cas=
e?  Also, has anyone studied the impact that working with very large JSON b=
odies has on the soft real-time properties of the Erlang VM?  Do you know w=
hat kind of throughput you see in terms of MB/sec? I guess if that number i=
s large enough then any reasonably-sized JSON body (even a very big batch o=
f KV pairs returned from a map_docs call) will be processed without mucking=
 up the scheduling too much.

> Adding a NIF based JSON decoding/encoding module
> ------------------------------------------------
>
>                 Key: COUCHDB-1118
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1118
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Filipe Manana
>             Fix For: 1.2
>
>
> Currently, all the Erlang based JSON encoders and decoders are very slow,=
 and decoding and encoding JSON is something that we do basically everywher=
e.
> Via IRC, it recently discussed about adding a JSON NIF encoder/decoder. D=
amien also started a thread at the development mailing list about adding NI=
Fs to trunk.
> The patch/branch at [1] adds such a JSON encoder/decoder. It is based on =
Paul Davis' eep0018 project [2]. Damien made some modifications [3] to it m=
ostly to add support for big numbers (Paul's eep0018 limits the precision t=
o 32/64 bits) and a few optimizations. I made a few corrections and minor e=
nhancements on top of Damien's fork as well [4]. Finally Beno=C3=AEt identi=
fied some missing capabilities compared to mochijson2 (on encoding, allow a=
toms as strings and strings as object properties).
> Also, the version added in the patch at [1] uses mochijson2 when the C NI=
F is not loaded. Autotools configuration was adapted to compile the NIF onl=
y when we're using an OTP release >=3D R13B04 (R13B03 NIF API is too limite=
d and suffered many changes compared to R13B04 and R14) - therefore it shou=
ld work on any OTP release > R13B at least.
> I successfully tested this on R13B03, R13B04 and R14B02 in an Ubuntu envi=
ronment.
> I'm not sure if it builds at all on Windows - would appreciate if someone=
 could verify it.
> Also, I'm far from being good with the autotools, so I probably missed so=
mething important or I'm doing something in a not very standard way.
> This NIF encoder/decoder is about one order of magnitude faster compared =
to mochijson2 and other Erlang-only solutions such as jsx. A read and write=
s test with relaximation shows this has a very positive impact, specially o=
n reads (the EJSON encoding is more expensive than JSON decoding) - http://=
graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef634052381
> @Paul, since this is based on your eep0018 effort, do you think any other=
 missing files should be added (README, etap tests, etc)? Also, should we p=
ut somewhere a note this is based on your project?
> [1] - https://github.com/fdmanana/couchdb/compare/json_nif
> [2] - https://github.com/davisp/eep0018
> [3] - https://github.com/Damienkatz/eep0018/commits/master
> [4] - https://github.com/fdmanana/eep0018/commits/final_damien

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira