couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Orr <nicholas....@zxgen.net>
Subject Re: Hanging replication (was Re: Replication Status Messages)
Date Wed, 04 Nov 2009 03:07:02 GMT
Awesome - will make sure I Pull ;)

Thanks,

Nick

On Wed, Nov 4, 2009 at 2:04 PM, Adam Kocoloski <kocolosk@apache.org> wrote:

> Hi Nick, pulling attachments should be much better.  It's able to use the
> standalone attachment API and thus avoid the Base64 encoding overhead of the
> push approach.  The data is streamed directly to the DB file, so the whole
> attachment never needs to fit in memory.  Best,
>
> Adam
>
>
> On Nov 3, 2009, at 9:56 PM, Nicholas Orr wrote:
>
>  What about pulling attachments?
>>
>> I'm getting real close to shoving attachments into my docs and would like
>> to
>> be able to replicate the docs with attachments....
>>
>> Nick
>>
>> On Wed, Nov 4, 2009 at 12:47 PM, Adam Kocoloski <kocolosk@apache.org>
>> wrote:
>>
>>  Thanks Matt & Matt, we should put a big warning sticker on the replicator
>>> saying "don't push attachments" :(  Getting push replication to
>>> efficiently
>>> handle attachments is a blocker for the next release as far as I'm
>>> concerned.  Best,
>>>
>>> Adam
>>>
>>>
>>> On Nov 3, 2009, at 7:58 PM, Matt Aimonetti wrote:
>>>
>>> I also experienced the same problems with replication + attachments.
>>>
>>>>
>>>> - Matt
>>>>
>>>> On Tue, Nov 3, 2009 at 4:51 PM, Matt Goodall <matt.goodall@gmail.com>
>>>> wrote:
>>>>
>>>> 2009/11/3 Adam Kocoloski <kocolosk@apache.org>:
>>>>
>>>>>
>>>>>  Yeah, that's too long, something seems to have gone south.  If
>>>>>> possible
>>>>>>
>>>>>>  I'd
>>>>>
>>>>>  try restarting CouchDB and then the replication.  Best,
>>>>>>
>>>>>>
>>>>> I've also been experiencing similar problems with replication hanging
>>>>> recently so I set about reproducing it tonight.
>>>>>
>>>>> In the end it was quite simple. I was fairly sure it was related to
>>>>> attachments so I created a database with a bunch of documents each
>>>>> containing a largish attachment. Then, I replicated that to an empty
>>>>> database on the same couchdb server but over HTTP.
>>>>>
>>>>> See attached shell script. Run with an init arg first time to setup a
>>>>> source database; run without the init arg on subsequent runs to avoid
>>>>> creating the source database. I'm running on Ubuntu 9.10 and the
>>>>> latest CouchDB 0.10.x branch.
>>>>>
>>>>> Replication starts ok but then hangs. The status page does not update
>>>>> and I start seeing "Uncaught error in HTTP request" errors in the
>>>>> couch logs.
>>>>>
>>>>> [error] [<0.1236.0>] Uncaught error in HTTP request: {exit,normal}
>>>>> [info] [<0.1236.0>] Stacktrace: [{mochiweb_request,send,2},
>>>>>         {mochiweb_request,respond,2},
>>>>>         {couch_httpd,send_response,4},
>>>>>         {couch_httpd_db,do_db_req,2},
>>>>>         {couch_httpd,handle_request,5},
>>>>>         {mochiweb_http,headers,5},
>>>>>         {proc_lib,init_p_do_apply,3}]
>>>>>
>>>>> CPU is typically close to 100% anyway but after the first error the
>>>>> erlang process's memory usage rises significantly.
>>>>>
>>>>> The only way to kill the hung replication process seems to be to
>>>>> restart the server.
>>>>>
>>>>> Interestingly, continuous replication seems to have the same problem
>>>>> but recovers from the failure and eventually completes.
>>>>>
>>>>> - Matt
>>>>>
>>>>>
>>>>>
>>>>>  Adam
>>>>>>
>>>>>> On Nov 3, 2009, at 3:37 PM, Zachary Zolton wrote:
>>>>>>
>>>>>> Coolio, so how long should I wait for that status message to change
>>>>>>
>>>>>>> locally before I start fretting? It's been the same for like
a good
>>>>>>> 10
>>>>>>> minutes, though I'm pushing to a remote couchy host on the
>>>>>>> intarwebs....
>>>>>>>
>>>>>>> (FYI, my local database is 7.1MB compacted, has 756 docs and
is at
>>>>>>> seq#
>>>>>>> 1318)
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 3, 2009 at 2:32 PM, Adam Kocoloski <kocolosk@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Nov 3, 2009, at 3:23 PM, Zachary Zolton wrote:
>>>>>>>>
>>>>>>>> Howdy,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've been watching a push replication in Futon that's
been
>>>>>>>>> crunching
>>>>>>>>> for a while with the following message:
>>>>>>>>> "W Processed source update #176"
>>>>>>>>>
>>>>>>>>> Does this mean that the remote server has process my
local
>>>>>>>>> revisions
>>>>>>>>> up to sequence number 176? Also, should I be able to
see an active
>>>>>>>>> task, or something on the replication page, for the remote
server?
>>>>>>>>>
>>>>>>>>> Any explanations of what to look for would be greatly
appreciated!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Zach
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi Zach, that's correct, everything up to source update #176
should
>>>>>>>> be
>>>>>>>> available on the target server at that point.
>>>>>>>>
>>>>>>>> There's no easy way to generate a status update on the target
>>>>>>>> server,
>>>>>>>> because to the target the replication just looks like a series
of
>>>>>>>>
>>>>>>>>  normal
>>>>>>>
>>>>>>
>>>>>  HTTP requests.  Cheers,
>>>>>>
>>>>>>>
>>>>>>>> Adam
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message