incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Hanging replication (was Re: Replication Status Messages)
Date Wed, 04 Nov 2009 03:04:16 GMT
Hi Nick, pulling attachments should be much better.  It's able to use  
the standalone attachment API and thus avoid the Base64 encoding  
overhead of the push approach.  The data is streamed directly to the  
DB file, so the whole attachment never needs to fit in memory.  Best,

Adam

On Nov 3, 2009, at 9:56 PM, Nicholas Orr wrote:

> What about pulling attachments?
>
> I'm getting real close to shoving attachments into my docs and would  
> like to
> be able to replicate the docs with attachments....
>
> Nick
>
> On Wed, Nov 4, 2009 at 12:47 PM, Adam Kocoloski  
> <kocolosk@apache.org> wrote:
>
>> Thanks Matt & Matt, we should put a big warning sticker on the  
>> replicator
>> saying "don't push attachments" :(  Getting push replication to  
>> efficiently
>> handle attachments is a blocker for the next release as far as I'm
>> concerned.  Best,
>>
>> Adam
>>
>>
>> On Nov 3, 2009, at 7:58 PM, Matt Aimonetti wrote:
>>
>> I also experienced the same problems with replication + attachments.
>>>
>>> - Matt
>>>
>>> On Tue, Nov 3, 2009 at 4:51 PM, Matt Goodall  
>>> <matt.goodall@gmail.com>
>>> wrote:
>>>
>>> 2009/11/3 Adam Kocoloski <kocolosk@apache.org>:
>>>>
>>>>> Yeah, that's too long, something seems to have gone south.  If  
>>>>> possible
>>>>>
>>>> I'd
>>>>
>>>>> try restarting CouchDB and then the replication.  Best,
>>>>>
>>>>
>>>> I've also been experiencing similar problems with replication  
>>>> hanging
>>>> recently so I set about reproducing it tonight.
>>>>
>>>> In the end it was quite simple. I was fairly sure it was related to
>>>> attachments so I created a database with a bunch of documents each
>>>> containing a largish attachment. Then, I replicated that to an  
>>>> empty
>>>> database on the same couchdb server but over HTTP.
>>>>
>>>> See attached shell script. Run with an init arg first time to  
>>>> setup a
>>>> source database; run without the init arg on subsequent runs to  
>>>> avoid
>>>> creating the source database. I'm running on Ubuntu 9.10 and the
>>>> latest CouchDB 0.10.x branch.
>>>>
>>>> Replication starts ok but then hangs. The status page does not  
>>>> update
>>>> and I start seeing "Uncaught error in HTTP request" errors in the
>>>> couch logs.
>>>>
>>>> [error] [<0.1236.0>] Uncaught error in HTTP request: {exit,normal}
>>>> [info] [<0.1236.0>] Stacktrace: [{mochiweb_request,send,2},
>>>>          {mochiweb_request,respond,2},
>>>>          {couch_httpd,send_response,4},
>>>>          {couch_httpd_db,do_db_req,2},
>>>>          {couch_httpd,handle_request,5},
>>>>          {mochiweb_http,headers,5},
>>>>          {proc_lib,init_p_do_apply,3}]
>>>>
>>>> CPU is typically close to 100% anyway but after the first error the
>>>> erlang process's memory usage rises significantly.
>>>>
>>>> The only way to kill the hung replication process seems to be to
>>>> restart the server.
>>>>
>>>> Interestingly, continuous replication seems to have the same  
>>>> problem
>>>> but recovers from the failure and eventually completes.
>>>>
>>>> - Matt
>>>>
>>>>
>>>>
>>>>> Adam
>>>>>
>>>>> On Nov 3, 2009, at 3:37 PM, Zachary Zolton wrote:
>>>>>
>>>>> Coolio, so how long should I wait for that status message to  
>>>>> change
>>>>>> locally before I start fretting? It's been the same for like a  
>>>>>> good 10
>>>>>> minutes, though I'm pushing to a remote couchy host on the
>>>>>> intarwebs....
>>>>>>
>>>>>> (FYI, my local database is 7.1MB compacted, has 756 docs and is 

>>>>>> at seq#
>>>>>> 1318)
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 3, 2009 at 2:32 PM, Adam Kocoloski <kocolosk@apache.org

>>>>>> >
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> On Nov 3, 2009, at 3:23 PM, Zachary Zolton wrote:
>>>>>>>
>>>>>>> Howdy,
>>>>>>>>
>>>>>>>> I've been watching a push replication in Futon that's been
 
>>>>>>>> crunching
>>>>>>>> for a while with the following message:
>>>>>>>> "W Processed source update #176"
>>>>>>>>
>>>>>>>> Does this mean that the remote server has process my local
 
>>>>>>>> revisions
>>>>>>>> up to sequence number 176? Also, should I be able to see
an  
>>>>>>>> active
>>>>>>>> task, or something on the replication page, for the remote
 
>>>>>>>> server?
>>>>>>>>
>>>>>>>> Any explanations of what to look for would be greatly  
>>>>>>>> appreciated!
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Zach
>>>>>>>>
>>>>>>>
>>>>>>> Hi Zach, that's correct, everything up to source update #176
 
>>>>>>> should be
>>>>>>> available on the target server at that point.
>>>>>>>
>>>>>>> There's no easy way to generate a status update on the target
 
>>>>>>> server,
>>>>>>> because to the target the replication just looks like a series
 
>>>>>>> of
>>>>>>>
>>>>>> normal
>>>>
>>>>> HTTP requests.  Cheers,
>>>>>>>
>>>>>>> Adam
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>


Mime
View raw message