couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Katz <dam...@apache.org>
Subject Re: [jira] Updated: (COUCHDB-160) replication performance improvements
Date Tue, 11 Nov 2008 20:56:58 GMT
Wow. Very cool!

One thing that was a problem in the past when attempting to make  
things more parallel was with process queues getting backed up. On  
large replications, if the write target was slow, the readers would be  
much faster than the writers and the write queue would get huge and it  
could cause the erlang vm memory usage to skyrocket, slowing  
everything else down and sometimes crashing. As a temporary fix the  
current replicator only keeps a single doc queued up at a time.

I've not closely at this yet. Is there any thing in this  
implementation that would exhibit similar behaviors if something gets  
behind, or the number of documents is huge?

-Damien


On Nov 11, 2008, at 1:55 PM, Adam Kocoloski (JIRA) wrote:

>
>     [ https://issues.apache.org/jira/browse/COUCHDB-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

>  ]
>
> Adam Kocoloski updated COUCHDB-160:
> -----------------------------------
>
>    Attachment: couch_rep.erl.diff
>
> should have mentioned before -- the times quoted are in seconds.
>
>> replication performance improvements
>> ------------------------------------
>>
>>                Key: COUCHDB-160
>>                URL: https://issues.apache.org/jira/browse/COUCHDB-160
>>            Project: CouchDB
>>         Issue Type: Improvement
>>         Components: Database Core
>>   Affects Versions: 0.9
>>           Reporter: Adam Kocoloski
>>           Priority: Minor
>>        Attachments: couch_rep.erl.diff
>>
>>
>> I wrote some code to speed up CouchDB's replication process by  
>> parallelizing document requests and using _bulk_docs to write  
>> changes to the target.  I tested the speedup as follows:
>> * 1000 document DB, 1022 update_seq, ~450 KB after compaction
>> * local and remote machines have ~45 ms latency
>> * timed requests using timer:tc(couch_rep, replicate,  
>> [<<"source">>, <<"target">>]
>> * all replications are "from scratch"
>> trunk:
>> local-local     115
>> local-remote    145
>> remote-remote   173
>> remote-local    146
>> db size after replication: 1.8 MB
>> patch:
>> local-local     1.83
>> local-remote    38
>> remote-remote   64
>> remote-local    35
>> db size after replication: 453 KB
>> I'll attach the patch as an update to this issue.  It might be  
>> worth exposing the "batch size" (currently 100 docs) as a  
>> configurable parameter.  Comments welcome.  Best,
>> Adam
>
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>


Mime
View raw message