couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Joseph Davis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COUCHDB-1118) Adding a NIF based JSON decoding/encoding module
Date Mon, 04 Apr 2011 20:39:05 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015609#comment-13015609
] 

Paul Joseph Davis commented on COUCHDB-1118:
--------------------------------------------

Just to point out, of the other four, one is mochijson packaged by itself. One is jsx which
focuses on decoding JSON streams into tokens like our json_stream_thinger and Alisdair said
he hasn't focused at all on encoding which more or less prompted this discussion. ktuo appears
to be pretty close to mochijson but a bit cleaner on the source side though I see its not
doing any sort of unicode handling which we spent so much time on. And lastly jsonerl is basically
a modified mochijson which redefines objects to be {{k1, v1}, {k2, v2}} which would break
our codes.

Other than that, I'd say I need a definition of measurements for what better is. Can a NIF
be much faster than all of the other implementations? Probably. But then some people wouldn't
want to touch it because its a NIF and not pure Erlang. On the other hand a NIF could have
an Erlang implementation as well. I'd probably focus on making a customized version of mochijson2
to ensure that both versions give the exact same output for a given input.

Also, whether such a project becomes *the* Erlang JSON package isn't too much of a concern
to me. Sure it'd be nice to see lots of people using our version so that we know its been
battle tested, but with the number of couchers out there beating on this public facing code,
I'm not too concerned that we'll fail to uncover any bugs quite quickly.

> Adding a NIF based JSON decoding/encoding module
> ------------------------------------------------
>
>                 Key: COUCHDB-1118
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1118
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Filipe Manana
>             Fix For: 1.2
>
>
> Currently, all the Erlang based JSON encoders and decoders are very slow, and decoding
and encoding JSON is something that we do basically everywhere.
> Via IRC, it recently discussed about adding a JSON NIF encoder/decoder. Damien also started
a thread at the development mailing list about adding NIFs to trunk.
> The patch/branch at [1] adds such a JSON encoder/decoder. It is based on Paul Davis'
eep0018 project [2]. Damien made some modifications [3] to it mostly to add support for big
numbers (Paul's eep0018 limits the precision to 32/64 bits) and a few optimizations. I made
a few corrections and minor enhancements on top of Damien's fork as well [4]. Finally BenoƮt
identified some missing capabilities compared to mochijson2 (on encoding, allow atoms as strings
and strings as object properties).
> Also, the version added in the patch at [1] uses mochijson2 when the C NIF is not loaded.
Autotools configuration was adapted to compile the NIF only when we're using an OTP release
>= R13B04 (R13B03 NIF API is too limited and suffered many changes compared to R13B04 and
R14) - therefore it should work on any OTP release > R13B at least.
> I successfully tested this on R13B03, R13B04 and R14B02 in an Ubuntu environment.
> I'm not sure if it builds at all on Windows - would appreciate if someone could verify
it.
> Also, I'm far from being good with the autotools, so I probably missed something important
or I'm doing something in a not very standard way.
> This NIF encoder/decoder is about one order of magnitude faster compared to mochijson2
and other Erlang-only solutions such as jsx. A read and writes test with relaximation shows
this has a very positive impact, specially on reads (the EJSON encoding is more expensive
than JSON decoding) - http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef634052381
> @Paul, since this is based on your eep0018 effort, do you think any other missing files
should be added (README, etap tests, etc)? Also, should we put somewhere a note this is based
on your project?
> [1] - https://github.com/fdmanana/couchdb/compare/json_nif
> [2] - https://github.com/davisp/eep0018
> [3] - https://github.com/Damienkatz/eep0018/commits/master
> [4] - https://github.com/fdmanana/eep0018/commits/final_damien

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message