couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antony Blakey <>
Subject Re: Attachment Replication Problem - Bug Found
Date Sat, 16 May 2009 14:39:13 GMT

On 16/05/2009, at 8:45 PM, Adam Kocoloski wrote:

> I encountered this "one more byte" problem once before, but it  
> wasn't 100% reproducible, so I wasn't really comfortable checking in  
> a workaround.  I've basically been waiting to see if it would ever  
> show up for anyone else :-/

I'm lucky to have a repeatable failure :/

> I think we should commit this change, but I'd still like to confirm  
> that the attachment on the target is not corrupted by the chunk  
> processing issue (i.e. the last chunk starts with a \r or something  
> like that).

I can confirm that the target and source of replicated resources  
affected by this issue are identical with this fix, and both are  
correct i.e. uncorrupted, although this is only according to the  
failures I've seen.

> Now, on to the checkpointing conditions.  I think there's some  
> confusion about the attachment workflow.  Attachments are downloaded  
> _immediately_ and in their entirety by ibrowse, which then sends the  
> data as 1MB binary chunks to the attachment receiver processes.

Are they downloaded to disk by ibrowse?

> In another thread Matt Goodall suggested checkpointing after a  
> certain amount of time has passed.  So we'd have a checkpointing  
> algo that considers
> * memory utilization
> * number of pending writes
> * time elapsed

That seems to cover both resource usage and incremental progress. As  
far as the couch_util:should_flush mechanism is concerned, I think a  
good idea would be to commit 1 document, then 2, then 4 i.e. a binary  
increasing window which adapts well to both unreliable and reliable  
connections without requiring configuration, which is tricky because  
you may want to run the system in a variety of scenarios, and you  
might not know what the failure characteristics are (and they may  
change over time).

While we on this - any idea about why couchdb is quiting during  
replication? It's not giving me any errors.

Antony Blakey
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Hi, I'd like to do $THING. I know that $SOLUTION_A and $SOLUTION_B  
will do it very easily and for a very reasonable price, but I don't  
want to use $SOLUTION_A or $SOLUTION_B because $VAGUE_REASON and  
$CONTRADICTORY_REASON. Instead, I'd like your under-informed ideas on  
how to achieve my $POORLY_CONCEIVED_AMBITIONS using Linux, duct tape,  
an iPod, and hours and hours of my precious time.
   -- Slashdot response to an enquiry

View raw message