couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <>
Subject [jira] Commented: (COUCHDB-1092) Storing documents bodies as raw JSON binaries instead of serialized JSON terms
Date Fri, 18 Mar 2011 13:51:29 GMT


Filipe Manana commented on COUCHDB-1092:

Ok, I tested Paul's branch and here are the results I got:

# View generation

$ time curl http://localhost:5984/branch_floats_zip_db/_design/test/_view/simple?limit=1

real	5m54.197s
user	0m0.000s
sys	0m0.012s

$ rm -fr couchdb/tmp/lib/.branch_floats_zip_db_design
$ echo 3 > /proc/sys/vm/drop_caches
$ time curl http://localhost:5984/branch_floats_zip_db/_design/test/_view/simple?limit=1

real	5m52.879s
user	0m0.004s
sys	0m0.016s

$ rm -fr couchdb/tmp/lib/.branch_floats_zip_db_design
$ echo 3 > /proc/sys/vm/drop_caches
$ time curl http://localhost:5984/branch_floats_zip_db/_design/test/_view/simple?limit=1

real	5m59.737s
user	0m0.000s
sys	0m0.016s

With my original branch, this takes about 4 minutes, with trunk it takes about 15 minutes.

I also ran relaximation several times, with delayed_commits set to false, and here are 2 of
those runs:

I think it's easy to see that both reads and writes are worse then trunk.

Also, apart from the vhosts and 173-os-daemon-cfg-register.t etap tests, I don't get any tests
failing. I think both are known to be failing in trunk, as several people have the same issue.

I mentioned several times this branch still has some TODOs and is not finished, and I told
it in the very first comment, I thought I was clear about that. Namely I plan on removing
the functions couch_doc:from_json_obj and couch_doc:to_json_obj, so that only the new ones
I added are used, since they are there also to guarantee correct behaviour. Part of the reason
to present a not yet finished and fully polished patch was to get feedback and stimulate the

Also, I haven't had my last questions answered by Paul.

1) Paul claims his jsonsplice addition helps preventing us getting invalid document bodies
getting written to disk, while what I see is that his addition does validation on the document
read operations path only

2) Besides that, our current code doesn't ensure that when someone calls one of the couch_doc:update_doc
or couch_doc:update_docs, the document bodies are EJSON objects. I have given an example in
a CouchDB trunk shell that shows this:

Therefore, this issue is not introduced by this patch/branch

3) These document update functions still accept EJSON document bodies like before, and I added
hooks to make sure they get converted to JSON binaries - therefore either users or developers
using the couch_db API, don't need to change they're code at all.

Two new Etap tests were added, exclusively dedicated to the new functions that were added
to the couch_doc module, I intended to remove the tests 030-doc-from-json.t and 031-doc-to-json.t
since they test the "old" functions couch_doc:from_json_obj and couch_doc:to_json_obj, that
I planned to remove from couch_doc, as I pointed before in this comment. Adam's suggestion
is not integrated yet as well.

I hope my comments and points are more clear now.

> Storing documents bodies as raw JSON binaries instead of serialized JSON terms
> ------------------------------------------------------------------------------
>                 Key: COUCHDB-1092
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>            Reporter: Filipe Manana
>            Assignee: Filipe Manana
> Currently we store documents as Erlang serialized (via the term_to_binary/1 BIF) EJSON.
> The proposed patch changes the database file format so that instead of storing serialized
> EJSON document bodies, it stores raw JSON binaries.
> The github branch is at:
> Advantages:
> * what we write to disk is much smaller - a raw JSON binary can easily get up to 50%
>   (at least according to the tests I did)
> * when serving documents to a client we no longer need to JSON encode the document body
>   read from the disk - this applies to individual document requests, view queries with
>   ?include_docs=true, pull and push replications, and possibly other use cases.
>   We just grab its body and prepend the _id, _rev and all the necessary metadata fields
>   (this is via simple Erlang binary operations)
> * we avoid the EJSON term copying between request handlers and the db updater processes,
>   between the work queues and the view updater process, between replicator processes,
> * before sending a document to the JavaScript view server, we no longer need to convert
>   from EJSON to JSON
> The changes done to the document write workflow are minimalist - after JSON decoding
> document's JSON into EJSON and removing the metadata top level fields (_id, _rev, etc),
> JSON encodes the resulting EJSON body into a binary - this consumes CPU of course but
> brings 2 advantages:
> 1) we avoid the EJSON copy between the request process and the database updater process
>    for any realistic document size (4kb or more) this can be very expensive, specially
>    when there are many nested structures (lists inside objects inside lists, etc)
> 2) before writing anything to the file, we do a term_to_binary([Len, Md5, TheThingToWrite])
>    and then write the result to the file. A term_to_binary call with a binary as the
>    is very fast compared to a term_to_binary call with EJSON as input (or some other
>    structure)
> I think both compensate the JSON encoding after the separation of meta data fields and
non-meta data fields.
> The following relaximation graph, for documents with sizes of 4Kb, shows a significant
> performance increase both for writes and reads - especially reads.   
> I've also made a few tests to see how much the improvement is when querying a view, for
> first time, without ?stale=ok. The size difference of the databases (after compaction)
> also very significant - this change can reduce the size at least 50% in common cases.
> The test databases were created in an instance built from that experimental branch.
> Then they were replicated into a CouchDB instance built from the current trunk.
> At the end both databases were compacted (to fairly compare their final sizes).
> The databases contain the following view:
> {
>     "_id": "_design/test",
>     "language": "javascript",
>     "views": {
>         "simple": {
>             "map": "function(doc) { emit(doc.float1, doc.strings[1]); }"
>         }
>     }
> }
> ## Database with 500 000 docs of 2.5Kb each
> Document template is at:
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test.couch 
> 1996	couchdb/tmp/lib/disk_json_test.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test.couch 
> 2693	couchdb-trunk/tmp/lib/disk_ejson_test.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl http://localhost:5984/disk_json_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real	6m6.740s
> user	0m0.016s
> sys	0m0.008s
> $ time curl http://localhost:5985/disk_ejson_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real	15m41.439s
> user	0m0.012s
> sys	0m0.012s
> ## Database with 100 000 docs of 11Kb each
> Document template is at:
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test_11kb.couch
> 1185	couchdb/tmp/lib/disk_json_test_11kb.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> 2202	couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl http://localhost:5984/disk_json_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real	4m19.306s
> user	0m0.008s
> sys	0m0.004s
> $ time curl http://localhost:5985/disk_ejson_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real	18m46.051s
> user	0m0.008s
> sys	0m0.016s
> All in all, I haven't seen yet any disadvantage with this approach. Also, the code changes
> don't bring additional complexity. I say the performance and disk space gains it gives
> very positive.
> This branch still needs to be polished in a few places. But I think it isn't far from
getting mature.
> Other experiments that can be done are to store view values as raw JSON binaries as well
(instead of EJSON)
> and optional compression of the stored JSON binaries (since it's pure text, the compression
ratio is very high).
> However, I would prefer to do these other 2 suggestions in separate branches/patches
- I haven't actually tested
> any of them yet, so maybe they not bring significant gains.
> Thoughts? :)

This message is automatically generated by JIRA.
For more information on JIRA, see:

View raw message