couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <>
Subject [jira] Commented: (COUCHDB-639) Make replication profit of attachment compression and improve push replication for large attachments
Date Fri, 26 Feb 2010 13:06:27 GMT


Filipe Manana commented on COUCHDB-639:

Just tested it with the latest trunk rev and found no problems:

fdmanana@core2duo:~/git/couchdb$ git log -1
commit 4cefde131f1992c70f66c527435a715290174423
Author: Mark Hammond <>
Date:   Fri Feb 26 01:32:21 2010 +0000

    generate .sha file for windows binary; ensure md5/sha use rel paths
    git-svn-id: 13f79535-47bb-0310-9956-ffa450edef68

fdmanana@core2duo:~/git/couchdb$ git apply --index --reject ../rep-att-comp-and-multipart-trunk-4.patch
Checking patch src/couchdb/couch_db.erl...
Checking patch src/couchdb/couch_doc.erl...
Checking patch src/couchdb/couch_httpd_db.erl...
Checking patch src/couchdb/couch_rep_att.erl...
Checking patch src/couchdb/couch_rep_reader.erl...
Checking patch src/couchdb/couch_rep_writer.erl...
Checking patch test/etap/170-replication-attachment-comp.t...
Checking patch test/etap/
Applied patch src/couchdb/couch_db.erl cleanly.
Applied patch src/couchdb/couch_doc.erl cleanly.
Applied patch src/couchdb/couch_httpd_db.erl cleanly.
Applied patch src/couchdb/couch_rep_att.erl cleanly.
Applied patch src/couchdb/couch_rep_reader.erl cleanly.
Applied patch src/couchdb/couch_rep_writer.erl cleanly.
Applied patch test/etap/170-replication-attachment-comp.t cleanly.
Applied patch test/etap/ cleanly.

All tests are passing also.

> Make replication profit of attachment compression and improve push replication for large
> ----------------------------------------------------------------------------------------------------
>                 Key: COUCHDB-639
>                 URL:
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Replication
>    Affects Versions: 0.11
>         Environment: trunk
>            Reporter: Filipe Manana
>         Attachments: rep-att-comp-and-multipart-trunk-4.patch
> At the moment, for compressed attachments, the replication uncompresses and then compresses
again the attachments. Therefore, a waste of CPU time.
> The push replication is also not reliable for very large attachments (500mb + for example).
Currently it sends the attachments in-lined in the respective JSON doc. Not only this requires
too much ram memory, it also wastes too much CPU time doing the base64 encoding of the attachment
(and also a decompression if the attachment is compressed).
> The following patch (rep-att-comp-and-multipart-trunk*.patch) addresses both issues.
Docs containing attachments are now streamed to the target remote DB using the multipart doc
streaming feature provided by couch_doc.erl, and compressed attachments are not uncompressed
and re-compressed during the replication
> JavaScript tests included.
> Previously doing a replication of a DB containing 2 docs with attachments of 100mb and
500mb caused the Erlang VM to consume near 1.2GB of ram memory in my system. With that patch
applied, it uses about 130Mb of ram memory.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message