couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COUCHDB-1120) Snappy compression (databases, view indexes) + keeping doc bodies as ejson binaries
Date Fri, 29 Apr 2011 20:47:03 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027175#comment-13027175
] 

Filipe Manana commented on COUCHDB-1120:
----------------------------------------

I've created another branch on top of the previous one which makes compression optional and
also adds the possibility to use deflate (zlib) compression:

https://github.com/fdmanana/couchdb/compare/file_compression

By default, snappy compression is enabled. The compression is configured in the section [couchdb]:
 https://github.com/fdmanana/couchdb/compare/file_compression#diff-4

For those interested, after checking out snappy from google code ( http://code.google.com/p/snappy/
), one can run some benchmark tests to compare snappy against zlib, lzo and other algorithms.
This is done by running:

$ ./snappy_unittest -run_microbenchmarks=false --zlib --lzo testdata/*

Output example at  http://friendpaste.com/7YVC8jImnY2GbnOLJvce6x

The tests I presented before, as well as the following ones, show that snappy has a positive
impact on the database read/write performance and view indexer performance.

Here are a few more tests against 2 different databases/views.


***** Database with 551 200 documents, each with a size of about 1 Kb *****

Database created with:

$ ./seatoncouch.rb --host localhost --port 5984 --docs 551200 --threads 20 --db-name complex_keys
\
	--bulk-batch 100 --doc-tpl complex_keys.tpl

The document template can be found here:  http://friendpaste.com/1cRdpfPyzWzoQKo8zb3fky

The database has following design document:

{
    "_id": "_design/test",
    "language": "javascript",
    "views": {
        "view1": {
            "map": "function(doc) { emit([doc.type, doc.category, doc.level, doc.ratio], doc.nested.coords);
}"
        },
        "view2": {
            "map": "function(doc) { emit(doc._id, {type: doc.type, cat: doc.category, level:
doc.level, ratio: doc.ratio}); }"
        }
    }
}



* trunk *

database file size after compaction:  1592 Mb
view file size after compaction:  520 Mb

view index build time:

$ time curl http://localhost:5984/trunk_complex_keys/_design/test/_view/view1?limit=1
{"total_rows":551200,"offset":0,"rows":[
{"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.73},{"x":153746.28,"y":190006.59}]}
]}

real	25m47.395s
user	0m0.008s
sys	0m0.040s


*** branch file_compression with snappy compression enabled ***

database file size after compaction:  1215 Mb
view file size after compaction:  155 Mb

view index build time:

$ time curl http://localhost:5985/snappy_complex_keys/_design/test/_view/view1?limit=1
{"total_rows":551200,"offset":0,"rows":[
{"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.73},{"x":153746.28,"y":190006.59}]}
]}

real	12m11.829s
user	0m0.004s
sys	0m0.020s


*** branch file_compression with deflate compression level 1 enabled ***

database file size after compaction: 1097 Mb
view file size after compaction:  123 Mb

view index build time:

$ time curl http://localhost:5985/deflate1_complex_keys/_design/test/_view/view1?limit=1
{"total_rows":551200,"offset":0,"rows":[
{"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.73},{"x":153746.28,"y":190006.59}]}
]}

real	19m32.945s
user	0m0.000s
sys	0m0.036s


*** branch file_compression with deflate compression level 9 enabled ***

database file size after compaction: 1092 Mb
view file size after compaction:  118 Mb

view index build time:

$ time curl http://localhost:5985/deflate9_complex_keys/_design/test/_view/view1?limit=1
{"total_rows":551200,"offset":0,"rows":[
{"id":"00d49881-7bcf-4c3d-a65d-e44435eeb513","key":["dwarf","assassin",2,1.1],"value":[{"x":174347.18,"y":127272.8},{"x":35179.93,"y":41550.55},{"x":157014.38,"y":172052.63},{"x":116185.83,"y":69871.73},{"x":153746.28,"y":190006.59}]}
]}

real	21m50.390s
user	0m0.012s
sys	0m0.036s





***** Benoit's warlogs database (https://warlogs.upondata.com/warlogs), 391 835 documents
*****

*** trunk ***

database file size after compaction: 1090 Mb
view file size after compaction:  1.3 Gb

view index build time:

$ time curl http://localhost:5984/warlogs_trunk/_design/warlogs/_view/by_date?limit=1
{"total_rows":391832,"offset":0,"rows":[
{"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
]}

real	44m10.341s
user	0m0.021s
sys	0m0.050s


*** branch file_compression with snappy compression enabled ***

database file size after compaction: 843 Mb
view file size after compaction:  616 Mb

view index build time:

$ time curl http://localhost:5985/warlogs_snappy/_design/warlogs/_view/by_date?limit=1
{"total_rows":391832,"offset":0,"rows":[
{"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
]}

real	32m53.988s
user	0m0.036s
sys	0m0.024s


*** branch file_compression with deflate compression level 1 enabled ***

database file size after compaction: 803 Mb
view file size after compaction:  459 Mb

view index build time:

time curl http://localhost:5985/warlogs_deflate1/_design/warlogs/_view/by_date?limit=1
{"total_rows":391832,"offset":0,"rows":[
{"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
]}

real	43m1.360s
user	0m0.008s
sys	0m0.068s


*** branch file_compression with deflate compression level 9 enabled ***

database file size after compaction: 798 Mb
view file size after compaction:  435 Mb

view index build time:

$ time curl http://localhost:5985/warlogs_deflate9/_design/warlogs/_view/by_date?limit=1
{"total_rows":391832,"offset":0,"rows":[
{"id":"0104D7FC-0219-4C16-8531-97C60A59C70C","key":["2004-01","0104D7FC-0219-4C16-8531-97C60A59C70C"],"value":null}
]}

real	47m10.841s
user	0m0.032s
sys	0m0.060s



I made several relaximation read and writes tests, as well as writes only tests (as shown
in the very first comment) and found out that both the reads and writes get a positive impact:


1) relaximation reads and writes test, 1 Kb documents (snappy compression vs trunk):

http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef6340655b7

(The 1 Kb document can be found here:  http://friendpaste.com/28yMMCXn5Dd0EPFpryrvMt


2) relaximation reads and writes test, 2.5 Kb documents (snappy compression vs trunk):

http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef634064d2d

(The 2.5 Kb document can be found here:  http://friendpaste.com/24dnXQT8FZ2gqGLI8571oV)


3) relaximation reads and writes test, 8 Kb documents (snappy compression vs trunk):

http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef634064f56

(The 8 Kb document can be found here:  http://friendpaste.com/2QN9TrpJA8476VzLzX0imy)


4) relaximation reads and writes test, 8 Kb documents (deflate level 1 compression vs trunk):

http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef634065e8f

With deflate level 1 compression, reads seem to stay about the same, while writes are still
slightly better compared to trunk.


I've also observed that in my machine, decompressing 100Kb with snappy is very fast, taking
between 100us and 150us, while zlib takes about ten times more for the same amount of data
 (which confirms the snappy benchmarks tests).

> Snappy compression (databases,  view indexes) + keeping doc bodies as ejson binaries
> ------------------------------------------------------------------------------------
>
>                 Key: COUCHDB-1120
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1120
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Database Core
>         Environment: trunk
>            Reporter: Filipe Manana
>            Assignee: Filipe Manana
>
> The branch at:
> https://github.com/fdmanana/couchdb/compare/snappy
> Is an experiment which adds snappy compression to database files and view index files.
Snappy is a very fast compressor/decompressor developed by and used by Google [1] - even for
small data chunks like 100Kb it can be 2 orders of magnitude faster then zlib or Erlang's
term_to_binary compression level 1. Somewhere at [1] there are benchmark results published
by Google that compare against zlib's deflate, Erlang's term_to_binary compression, lzo, etc.
> Even small objects like database headers or btree nodes, still get smaller after compressing
them with snappy, see the shell session at [2].
> Besides the compression, this branch also keeps the document bodies (#doc.body fields)
as binaries (snappy compressed ejson binaries) and only converts them back to ejson when absolutely
needed (done by couch_doc:to_json_obj/2 for e.g.) - this is similar to COUCHDB-1092 - but
the bodies are EJSON compressed binaries and doesn't suffer from the same issue Paul identified
before (which could be fixed without many changes) - on reads we decompress and still do the
binary_to_term/1 + ?JSON_ENCODE calls as before.
> It also prepares the document summaries before sending the documents to the updater,
so that we avoid copying EJSON terms and move this task outside of the updater to add more
parallelism to concurrent updates.
> I made some tests, comparing trunk before and after the JSON parser NIF was added, against
this snappy branch.
> I created databases with 1 000 000 documents of 4Kb each. The document template is this
one:  http://friendpaste.com/qdfyId8w1C5vkxROc5Thf
> The databases have this design document:
> {
>     "_id": "_design/test",
>     "language": "javascript",
>     "views": {
>         "simple": {
>             "map": "function(doc) { emit(doc.data5.float1, [doc.strings[2], doc.strings[10]]);
}"
>         }
>     }
> }
> == Results with trunk ==
> database file size after compaction:  7.5 Gb
> view index file size after compaction:  257 Mb
> ** Before JSON nif:
> $ time curl 'http://localhost:5985/trunk_db_1m/_design/test/_view/simple?limit=1'
> {"total_rows":1000000,"offset":0,"rows":[
> {"id":"00000632-d25d-49c6-9b4e-e038b78ff97d","key":76.572,"value":["jURcBZ0vrJcmf2roZUMzZJQoTsKZDIdj7KhO7itskKvM80jBU9","fKYYthv8iFvaYoFoYZyB"]}
> ]}
> real	58m28.599s
> user	0m0.036s
> sys	0m0.056s
> ** After JSON nif:
> fdmanana 12:45:55 /opt/couchdb > time curl 'http://localhost:5985/trunk_db_1m/_design/test/_view/simple?limit=1'
> {"total_rows":1000000,"offset":0,"rows":[
> {"id":"00000632-d25d-49c6-9b4e-e038b78ff97d","key":76.572,"value":["jURcBZ0vrJcmf2roZUMzZJQoTsKZDIdj7KhO7itskKvM80jBU9","fKYYthv8iFvaYoFoYZyB"]}
> ]}
> real	51m14.738s
> user	0m0.040s
> sys	0m0.044s
> == Results with the snappy branch ==
> database file size after compaction:  3.2 Gb   (vs 7.5 Gb on trunk)
> view index file size after compaction:  100 Mb  (vs 257 Mb on trunk)
> ** Before JSON nif:
> $ time curl 'http://localhost:5984/snappy_db_1m/_design/test/_view/simple?limit=1'
> {"total_rows":1000000,"offset":0,"rows":[
> {"id":"00000632-d25d-49c6-9b4e-e038b78ff97d","key":76.572,"value":["jURcBZ0vrJcmf2roZUMzZJQoTsKZDIdj7KhO7itskKvM80jBU9","fKYYthv8iFvaYoFoYZyB"]}
> ]}
> real	32m29.854s
> user	0m0.008s
> sys	0m0.052s
> ** After JSON nif:
> fdmanana 15:40:39 /opt/couchdb > time curl 'http://localhost:5984/snappy_db_1m/_design/test/_view/simple?limit=1'
> {"total_rows":1000000,"offset":0,"rows":[
> {"id":"00000632-d25d-49c6-9b4e-e038b78ff97d","key":76.572,"value":["jURcBZ0vrJcmf2roZUMzZJQoTsKZDIdj7KhO7itskKvM80jBU9","fKYYthv8iFvaYoFoYZyB"]}
> ]}
> real	18m39.240s
> user	0m0.012s
> sys	0m0.020s
> A writes-only relaximation test also shows a significant improvement in the writes response
times / throughput:
> http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef63405480d
> These results are also in a file of this branch [3].
> Seems clear this, together with Paul's JSON NIF parser, has a very good impact in the
view indexer, besides the big disk space savings and better write throughput.
> Some potential issues:
> * Snappy is C++, and so is the NIF [4] - however a C++ compiler is common and part of
most development environments (gcc, xcode, etc)
> * Not sure if snappy builds on Windows - it might build, it doesn't seem to depend on
fancy libraries, just stdc++ and the STL
> * Requires OTP R13B04 or higher. If built/running on R13B03 or below, it simple doesn't
do any compression at all, just like current releases. However, 2 servers running this branch,
one with R14 and other R13B01 for e.g., means that the second server will not be able to read
database files created by the server with R14 - it will get an exception with the atom 'snappy_nif_not_loaded'
- this is easy to catch and use for printing a nice and explicit error message to the user
telling it needs to use a more recent otp release.
> The upgrade of databases and view indexes from previous releases is done on compaction
- I made just a few tests with database files by hand, this surely needs to be better tested.
> Finally the branch is still in development phase, but maybe not far from completion,
consider this ticket just as a way to share some results and get some feedback.
> [1] - http://code.google.com/p/snappy/
> [2] - http://friendpaste.com/45AOdi9MkFrS4BPsov7Lg8
> [3] - https://github.com/fdmanana/couchdb/blob/b8f806e41727ba18ed6143cee31a3242e024ab2c/snappy-couch-tests.txt
> [4] - https://github.com/fdmanana/snappy-erlang-nif/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message