incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: Replacing ejson with jiffy
Date Wed, 01 Jun 2011 15:29:50 GMT
On Wed, Jun 1, 2011 at 8:34 AM, Filipe David Manana <fdmanana@apache.org> wrote:
> On Sat, May 28, 2011 at 8:38 PM, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>> Recently we added the ejson NIF to trunk to speedup JSON parsing. I've
>> finally put some finishing touches on a NIF that does much more of the
>> actual encoding and decoding in C while still maintaining the same
>> semantics as mochijson2. Specifically large numbers are still handled
>> without truncation among a few other things like unicode handling.
>
> Good work Paul.
>
>>
>> Things left to consider: I need to fall back to mochijson2 when jiffy
>> isn't available due to having an old Erlang.
>>
>> More importantly, Jiffy requires an even newer Erlang than Ejson. I
>> think R14B01, but it might even be R14B02. Not sure what people might
>> think of that either.
>
> Ideally I think R14B (< R14B01) should be supported. Is there really
> some significant NIF API changes in 14B01 onwards that can not be
> avoided? Anyway, I wouldn't make this a blocker.
>
>>
>> Thoughts?
>>
>
> Can you share your benchmark results, compared to latest trunk, and
> how have you tested it?
>
> I made a few basic tests today only for database writes only and view
> indexing. I used couch_http_writes.sh and couch_http_bulk_writes.sh
> from my github [1], both with 1Kb documents. I did not have the time
> yet to see what relaximation results are.
> Here they are:
>
> HTTP api, single document writes, 50 writers, 5 minutes run
> (couch_http_writes.sh)
>
>     jiffy
>
>     76 199 documents written in 5 minutes
>
>     trunk
>
>     79 513 documents written in 5 minutes
>
>
> HTTP api, bulk writes, 50 writers, batches of 10 docs, 5 minutes run
> (couch_http_bulk_writes.sh)
>
>     jiffy
>
>     616 960 documents written in 5 minutes
>
>     trunk
>
>     652 760 documents written in 5 minutes
>
>
> HTTP api, bulk writes, 50 writers, batches of 100 docs, 5 minutes run
> (couch_http_bulk_writes.sh)
>
>     jiffy
>
>     1 531 400 documents written in 5 minutes
>
>     trunk
>
>     1 314 200 documents written in 5 minutes
>
>
> Indexing Benoît's warlogs database (https://warlogs.upondata.com/warlogs):
>
> $ time curl http://localhost:5984/warlogs/_design/warlogs/_view/by_date?limit=1
> {"total_rows":391832,"offset":0,"rows":[
> {"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
> ]}
>
> real    34m39.351s
> user    0m0.028s
> sys     0m0.036s
>
> Takes roughly the same as trunk, variations of a few minutes (1, 2, 3)
> are normal for this view (in machine at least, and always after
> clearing the caches with "echo 3 > /proc/sys/vm/drop_caches")
>
>
> [1] - https://github.com/fdmanana/basho_bench_couch
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>

Filipe,

Awesome. I haven't gotten to testing it inside CouchDB yet other than
making sure tests pass. I was gonna try and do view build tests last
night but I wanted to find what database/view combination you've been
using. I'll try some tests against the warlogs db tonight and report
back.

The only testing I've done was in isolation with the various JSON
parsers. The code for those benchmarks is up at [1].


Thanks,
Paul

[1] https://github.com/davisp/erljson_bench

Mime
View raw message