couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Katz <>
Subject Re: replication usage? creating dupes?
Date Wed, 16 Jul 2008 17:06:00 GMT

On Jul 16, 2008, at 12:05 PM, Paul Davis wrote:

> I haven't really gotten into replication yet, but did I read that
> right? The browser request for compaction isn't expected to return
> until replication has completed? On the surface of things that seems
> fairly ungood. What happens in the future when I have a multi gigabyte
> database I want to replicate from scratch to a new node over a slow
> connection?
> If I'm not completely off my rocker, perhaps a better solution is that
> the browser request for replication returns immediately and then
> couchdb would provide a method for checking on the status of the
> replication.

That's an option for the future, but then we'd have to create a  
replication task monitor infrastructure that can be queried from HTTP.  
That's additional complexity and overhead.

If you want to fire and forget, then that's an easy option. But if you  
want to fire it off, and monitor it, and shut it down, then all that  
has to be written and tested and documented.

Instead, we can just have a synchronous HTTP request and get most of  
that for free. If the request is alive, you know the replication is  
still running. If you want to kill it, then terminate the connection.  
If you want to know when its done, then simply wait for the  
completion. We could even send updates about progress of the  
replication over chunked HTTP.

So for now I think we stick with the simpler design and enhance it to  
do what we want, until we hit the wall and need to build something  


> Feel free to ignore me if I have this completely wrong.
> Paul
> On Wed, Jul 16, 2008 at 11:58 AM, Damien Katz <>  
> wrote:
>> That problem is likely due to the fact the user HTTP request is  
>> timing out
>> while waiting for the replication to complete, that in turn kills the
>> underlying replication process. Restarting the replication will  
>> usually help
>> as CouchDB avoids sending the same document twice, but if the  
>> replication is
>> exceptionally long it might not get past the point where it it  
>> finishing
>> examining the documents.
>> The problem is its only saves off the replication record once it  
>> completes
>> successfully, so until it completes it always examine the same  
>> number of
>> documents to see if they exist on the target replica. The fix I  
>> need to
>> implement is to have it save off the replication record every x  
>> seconds
>> during replication, then if it dies unexpectedly it will pick back  
>> up from
>> the last replication record, reducing the number of documents  
>> needing to be
>> reexamined.
>> Then we need to solve is the current problem of synchronous HTTP  
>> request to
>> perform the replication. In Futon, the browser doesn't do the  
>> replication,
>> it just sends a single replication request to the CouchDB server. A  
>> CouchDB
>> Erlang process then performs the replication, accessing database  
>> either
>> locally or via HTTP on other Erlang servers. Right now, the browser  
>> can
>> timeout the HTTP request during a long replication, that in turn  
>> kills the
>> replication process.
>> There are two potential solutions here, the first is to send a  
>> browser ping
>> to keep the connection alive. Easy do do with HTTP 1.1 I think,  
>> just send an
>> empty HTTP chunk. The second is to make it impossible for the  
>> broken HTTP
>> request to kill the replication request. They aren't mutually  
>> exclusive, but
>> the more I think about it, the more I dislike the second solution.
>> -Damien
>> On Jul 16, 2008, at 11:13 AM, Chris Anderson wrote:
>>> On Wed, Jul 16, 2008 at 2:18 AM, Jan Lehnardt <>  
>>> wrote:
>>>> I'm surprised that his wasn't reported earlier. CouchDB replication
>>>> is supposed to be reliable (when we got all the bugs out), so an
>>>> external replication thing should not be necessary. I would have
>>>> guessed that reporting this is easier than writing code to  
>>>> circumvent
>>>> the problem. This should be fixed in CouchDB and not worked
>>>> around.
>>> My experience with replication has been that it works flawlessly for
>>> smaller datasets, and as the dataset grows, it either starts to take
>>> so long it may as well be broken (but shows no errors in the log) or
>>> occasionally does the =ERROR REPORT==== thing in the log. The  
>>> later is
>>> a new symptom in my experience.
>>> I haven't had a chance to bring my install up to latest trunk, so I
>>> hesitated to report it. Today's my only sane day for a couple of  
>>> weeks
>>> on each side, so I'll see what progress I can make.
>>> Chris
>>> --
>>> Chris Anderson

View raw message