couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Davis" <paul.joseph.da...@gmail.com>
Subject Re: replication usage? creating dupes?
Date Wed, 16 Jul 2008 16:05:21 GMT
I haven't really gotten into replication yet, but did I read that
right? The browser request for compaction isn't expected to return
until replication has completed? On the surface of things that seems
fairly ungood. What happens in the future when I have a multi gigabyte
database I want to replicate from scratch to a new node over a slow
connection?

If I'm not completely off my rocker, perhaps a better solution is that
the browser request for replication returns immediately and then
couchdb would provide a method for checking on the status of the
replication.

Feel free to ignore me if I have this completely wrong.

Paul

On Wed, Jul 16, 2008 at 11:58 AM, Damien Katz <damien@apache.org> wrote:
> That problem is likely due to the fact the user HTTP request is timing out
> while waiting for the replication to complete, that in turn kills the
> underlying replication process. Restarting the replication will usually help
> as CouchDB avoids sending the same document twice, but if the replication is
> exceptionally long it might not get past the point where it it finishing
> examining the documents.
>
> The problem is its only saves off the replication record once it completes
> successfully, so until it completes it always examine the same number of
> documents to see if they exist on the target replica. The fix I need to
> implement is to have it save off the replication record every x seconds
> during replication, then if it dies unexpectedly it will pick back up from
> the last replication record, reducing the number of documents needing to be
> reexamined.
>
> Then we need to solve is the current problem of synchronous HTTP request to
> perform the replication. In Futon, the browser doesn't do the replication,
> it just sends a single replication request to the CouchDB server. A CouchDB
> Erlang process then performs the replication, accessing database either
> locally or via HTTP on other Erlang servers. Right now, the browser can
> timeout the HTTP request during a long replication, that in turn kills the
> replication process.
>
> There are two potential solutions here, the first is to send a browser ping
> to keep the connection alive. Easy do do with HTTP 1.1 I think, just send an
> empty HTTP chunk. The second is to make it impossible for the broken HTTP
> request to kill the replication request. They aren't mutually exclusive, but
> the more I think about it, the more I dislike the second solution.
>
> -Damien
>
>
> On Jul 16, 2008, at 11:13 AM, Chris Anderson wrote:
>
>> On Wed, Jul 16, 2008 at 2:18 AM, Jan Lehnardt <jan@apache.org> wrote:
>>>
>>> I'm surprised that his wasn't reported earlier. CouchDB replication
>>> is supposed to be reliable (when we got all the bugs out), so an
>>> external replication thing should not be necessary. I would have
>>> guessed that reporting this is easier than writing code to circumvent
>>> the problem. This should be fixed in CouchDB and not worked
>>> around.
>>
>> My experience with replication has been that it works flawlessly for
>> smaller datasets, and as the dataset grows, it either starts to take
>> so long it may as well be broken (but shows no errors in the log) or
>> occasionally does the =ERROR REPORT==== thing in the log. The later is
>> a new symptom in my experience.
>>
>> I haven't had a chance to bring my install up to latest trunk, so I
>> hesitated to report it. Today's my only sane day for a couple of weeks
>> on each side, so I'll see what progress I can make.
>>
>> Chris
>>
>>
>> --
>> Chris Anderson
>> http://jchris.mfdz.com
>
>

Mime
View raw message