couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Kocoloski (JIRA)" <j...@apache.org>
Subject [jira] Commented: (COUCHDB-270) Replication w/ Large Attachments Fails
Date Tue, 03 Mar 2009 04:18:56 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678231#action_12678231
] 

Adam Kocoloski commented on COUCHDB-270:
----------------------------------------

Hi Jeff, thanks for the test case -- it proved quite useful. I have a short-term fix and a
longer-term solution. I'll attach the quick fix now (still waiting on SVN commit access).
It merely sets the ibrowse timeout to infinity and does a better check of the memory utilization
when deciding whether to flush the save_docs_buffer. 

The longer term solution will likely require scrapping the parallelized portion of the replicator.
Processing 100 documents in parallel doesn't give us enough control over the memory utilization
on the servers. I'm confident I can rewrite the module so that it loads documents into memory
"just-in-time" and maintains the same throughput with a much lower memory footprint. 

I ran your test suite between two m1.large instances on EC2 and got the following results:


200x512k push OK 
200x512k pull OK 
200x1m push OK 
200x1m pull OK 
200x20m push OK 
200x20m pull FAIL 
  
The last one failed in "head shot" fashion -- I believe the target server Erlang VM ran out
of memory. Fixing that one will require the long-term solution. Will keep you posted, 

Adam

> Replication w/ Large Attachments Fails
> --------------------------------------
>
>                 Key: COUCHDB-270
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-270
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>    Affects Versions: 0.9
>         Environment: Apache CouchDB 0.9.0a748379
>            Reporter: Jeff Hinrichs
>         Attachments: couchdb270_Test.py
>
>
> Attempting to replicate a database with largish attachments (<= ~18MB of attachments
in a doc, less thatn 200 docs)  from one machine to another fails consistently and at the
same point.
> Scenario:
> Both servers are running from HEAD and I've been tracking for some time.  This problem
has been around as long as I've been using couch.
> Machine A holds the original database, Machine B is the server that is doing a PULL replication
> During the replication, Machine A starts showing the following sporadically in the log:
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5902.3>] 'GET'
> /delasco-invoices/INV00652429?revs=true&attachments=true&latest=true&open_revs=["425644723"]
> {1,
>                             1}
> Headers: [{'Host',"192.168.2.52:5984"}]
> [Fri, 27 Feb 2009 14:02:48 GMT] [error] [<0.5901.3>] Uncaught error in
> HTTP request: {exit,normal}
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5901.3>] Stacktrace:
> [{mochiweb_request,send,2},
>             {couch_httpd,send_chunk,2},
>             {couch_httpd_db,db_doc_req,3},
>             {couch_httpd_db,do_db_req,2},
>             {couch_httpd,handle_request,3},
>             {mochiweb_http,headers,5},
>             {proc_lib,init_p,5}]
> [Fri, 27 Feb 2009 14:02:48 GMT] [debug] [<0.5901.3>] HTTPd 500 error response:
>  {"error":"error","reason":"normal"}
> As the replication continues, the frequency of these error "Uncaught error in HTTP request:
{exit,normal}"  increase.  Until the error is being constantly repeated.  Then Machine B stops
sending requests, no more log output, no errors, the last thing in Machine B's log file is:
> [Fri, 27 Feb 2009 14:03:24 GMT] [info] [<0.20893.1>] retrying
> couch_rep HTTP get request due to {error, req_timedout}: [104,116,
>                                                                   116,112,58,
>                                                                   47,47,49,
>                                                                   57,50,46,
>                                                                   49,54,56,
>                                                                   46,50,46,
>                                                                   53,50,58,
>                                                                   53,57,56,
>                                                                   52,47,100,
>                                                                   101,108,97,
>                                                                   115,99,111,
>                                                                   45,105,110,
>                                                                   118,111,
>                                                                   105,99,101,
>                                                                   115,47,73,
>                                                                   78,86,48,
>                                                                   48,54,53,
>                                                                   50,49,51,
>                                                                   56,63,114,
>                                                                   101,118,
>                                                                   115,61,116,
>                                                                   114,117,
>                                                                   101,38,97,
>                                                                   116,116,97,
>                                                                   99,104,109,
>                                                                   101,110,
>                                                                   116,115,61,
>                                                                   116,114,
>                                                                   117,101,38,
>                                                                   108,97,116,
>                                                                   101,115,
>                                                                   116,61,116,
>                                                                   114,117,
>                                                                   101,38,111,
>                                                                   112,101,
>                                                                   110,95,114,
>                                                                   101,118,
>                                                                   115,61,91,
>                                                                   34,
> <<"3070455362">>,
>                                                                   34,93]
> A request for status from the couchdb init.d script returns nothing and checking the
processes returns:
> (demo-couchdb)jlh@mars:~/projects/venvs/demo-couchdb/src$ ps ax|grep cou
> 29281 pts/2    S+     0:00 grep cou
> (demo-couchdb)jlh@mars:~/projects/venvs/demo-couchdb/src$ ps ax|grep beam
> 29305 pts/2    R+     0:00 grep beam
> In fact, couch has gone away completely on Machine B.  In fact, couch's death is so quick
it can't even say why.
> Attempts to incrementally replicate after the first failure die at exactly the same place.
> I can replicate this same database on the same machine from one database to another without
issue.  I can dump and reload the database with no problems.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message