couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antony Blakey <antony.bla...@gmail.com>
Subject Re: Attachment Replication Problem
Date Fri, 15 May 2009 07:55:14 GMT

On 15/05/2009, at 2:44 PM, Antony Blakey wrote:

> I have a 3.5G Couchdb database, consisting of 1000 small documents,  
> each with many attachments (0-30 per document), each attachment  
> varying wildly in size (1K..10M).
>
> To test replication I am running a server on my MBPro and another  
> under Ubuntu in VMWare on the same machine. I'm testing using a pure  
> trunk.
>
> Doing a pull-replicate from OSX to Linux fails to complete. The  
> point at which it fails is constant. I've added some debug logs into  
> couch_rep/attachment_loop like this: http://gist.github.com/112070  
> and made the suggested "couch_util:should_flush(1000)" mod to try  
> and guarantee progress (but to no avail). The debug output shows  
> this: http://gist.github.com/112069 and the document it seems to  
> fail on is this: http://gist.github.com/112074 . I'm only just  
> starting to look at this - any pointers would be appreciated.

It's not immediately obvious to me at the moment, but I wonder if with  
many small documents with lots of attachments considerable larger than  
the document size, it's possible to spawn too many async attachment  
downloads? In which case, maybe throttling the number of concurrent  
downloads would be a good idea.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

The greatest challenge to any thinker is stating the problem in a way  
that will allow a solution
   -- Bertrand Russell


Mime
View raw message