Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7162C7B0D for ; Fri, 5 Aug 2011 08:01:56 +0000 (UTC) Received: (qmail 94407 invoked by uid 500); 5 Aug 2011 08:01:50 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 92831 invoked by uid 500); 5 Aug 2011 08:01:39 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 92819 invoked by uid 99); 5 Aug 2011 08:01:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 08:01:36 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [130.83.156.225] (HELO lnx500.hrz.tu-darmstadt.de) (130.83.156.225) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 08:01:26 +0000 Received: from mailserver3.hrz.tu-darmstadt.de (lnx504.hrz.tu-darmstadt.de [130.83.156.233]) by lnx500.hrz.tu-darmstadt.de (8.14.4/8.14.4/HRZ/PMX) with ESMTP id p75815fT020469 for ; Fri, 5 Aug 2011 10:01:05 +0200 (envelope-from bruch@cs.tu-darmstadt.de) Received: from [130.83.165.50] (helo=dhcp-1.st.informatik.tu-darmstadt.de) by mailserver3.hrz.tu-darmstadt.de with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1QpFL7-000204-4m for user@couchdb.apache.org; Fri, 05 Aug 2011 10:01:05 +0200 From: Marcel Bruch Content-Type: multipart/alternative; boundary=Apple-Mail-50--783384106 Subject: Replication fails with timeout on large database/large documents Date: Fri, 5 Aug 2011 10:01:04 +0200 Message-Id: <670B29E4-E5BA-4D7A-886F-D393954E2942@cs.tu-darmstadt.de> To: user@couchdb.apache.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) X-PMX-TU: seen v1.2 by 5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.8.5.75115 X-PMX-RELAY: outgoing X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-50--783384106 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I'm currently trying to get replication work for large databases. = Unfortunately, replication terminates after a small number of documents = with some timeouts. Replication for several other databases with smaller = documents and total db size worked w/o any problem.=20 Before termination a few documents have been published (46MB). The = documents itself are rather complex and the DB is currently 6.5G in = size. Any ideas what's going wrong or how to solve? Thanks, Marcel Log of the source server shows: [Fri, 05 Aug 2011 07:46:51 GMT] [info] [<0.8544.0>] 127.0.0.1 - - 'GET' = /udc/_design/metaData/_view/modelSpecificationsIncludeAlias?key=3D%22org.e= clipse.swtbot.eclipse.core%22 200 [Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1030.0>] couch_rep_httpc = request failed after 10 retries: http://137.mmm.mmm.mmm:5984/udc/ [Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1009.0>] ** Generic server = <0.1009.0> terminating=20 ** Last message in was {'EXIT',<0.1030.0>, {http_request_failed, <<"failed to replicate = http://137.mmm.mmm.mmm:5984/udc/">>}} ** When Server state =3D=3D = {state,<0.1024.0>,<0.1026.0>,<0.1028.0>,<0.1030.0>, {db,<0.256.0>,<0.257.0>,nil, = <<"1312529491031938">>,<0.254.0>,<0.258.0>, {db_header,5,166309,0, {6931331728,{163315,2988}}, {6931334614,166303}, {6931341991,[]}, 0,nil,nil,1000}, 166309, {btree,<0.254.0>, {6931331728,{163315,2988}}, #Fun, #Fun, #Fun, #Fun}, {btree,<0.254.0>, {6931334614,166303}, #Fun, #Fun, #Fun, = #Fun}, {btree,<0.254.0>, {6931341991,[]}, #Fun, #Fun, #Fun,nil}, 166309,<<"udc">>, = "/usr/local/var/lib/couchdb/udc.couch",[],[], nil, {user_ctx,null, [<<"_admin">>], <<"{couch_httpd_auth, = default_authentication_handler}">>}, nil,1000, = [before_header,after_header,on_file_open], false}, = {http_db,"http://137.mmm.mmm.mmm:5984/udc/",[], [], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, false,false, ["f74fbcf318ccd6c8e53c0ffa0b679e74", {[{<<"source">>,<<"udc">>}, {<<"target">>, = <<"http://137.mmm.mmm.mmm:5984/udc/">>}]}, {user_ctx,null, [<<"_admin">>], <<"{couch_httpd_auth, = default_authentication_handler}">>}], = nil,0,[],<<"9d8ddc375c3ca782ead6cbc84d6e611a">>, {doc, = <<"_local/f74fbcf318ccd6c8e53c0ffa0b679e74">>, {0,[<<"6">>]}, {[{<<"session_id">>, = <<"3d35ed9a1de0e9c66135b54b55971d2c">>}, {<<"source_last_seq">>,732}, {<<"replication_id_version">>,2}, {<<"history">>, [{[{<<"session_id">>, = <<"3d35ed9a1de0e9c66135b54b55971d2c">>}, {<<"start_time">>, <<"Fri, 05 Aug 2011 07:28:50 = GMT">>}, {<<"end_time">>, <<"Fri, 05 Aug 2011 07:29:05 = GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,732}, {<<"recorded_seq">>,732}, {<<"missing_checked">>,0}, {<<"missing_found">>,4005}, {<<"docs_read">>,2128}, {<<"docs_written">>,733}, = {<<"doc_write_failures">>,0}]}]}]}, [],false,[]}, {doc, = <<"_local/f74fbcf318ccd6c8e53c0ffa0b679e74">>, {0,[<<"2">>]}, {[]}, [],false,[]}, "Fri, 05 Aug 2011 07:32:54 GMT", = <<"1312529491031938">>,<<"1312530889632808">>, {[{<<"session_id">>, <<"9d8ddc375c3ca782ead6cbc84d6e611a">>}, {<<"source_last_seq">>,1056}, {<<"replication_id_version">>,2}, {<<"history">>, [{[{<<"session_id">>, = <<"9d8ddc375c3ca782ead6cbc84d6e611a">>}, {<<"start_time">>, <<"Fri, 05 Aug 2011 07:32:54 GMT">>}, {<<"end_time">>, <<"Fri, 05 Aug 2011 07:33:10 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,1056}, {<<"recorded_seq">>,1056}, {<<"missing_checked">>,0}, {<<"missing_found">>,4004}, {<<"docs_read">>,2812}, {<<"docs_written">>,1057}, {<<"doc_write_failures">>,0}]}]}]}, [{<0.4586.0>,#Ref<0.0.0.36762>}, {<0.3631.0>,#Ref<0.0.0.35475>}, {<0.137.0>,#Ref<0.0.0.4775>}], false,1056,65587, {[{<<"source">>,<<"udc">>}, {<<"target">>, = <<"http://137.mmm.mmm.mmm:5984/udc/">>}]}, <0.1031.0>,nil} ** Reason for termination =3D=3D=20 ** {http_request_failed,<<"failed to replicate = http://137.mmm.mmm.mmm:5984/udc/">>} [Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1009.0>] = {error_report,<0.32.0>, {<0.1009.0>,crash_report, = [[{initial_call,{couch_rep,init,['Argument__1']}}, {pid,<0.1009.0>}, {registered_name,[]}, {error_info, {exit, {http_request_failed, <<"failed to replicate = http://137.mmm.mmm.mmm:5984/udc/">>}, [{gen_server,terminate,6}, {proc_lib,init_p_do_apply,3}]}}, {ancestors, [couch_rep_sup,couch_primary_services, couch_server_sup,<0.33.0>]}, {messages,[]}, {links, [<0.1028.0>,<0.1031.0>,<0.1024.0>,<0.1026.0>, <0.83.0>]}, {dictionary, = [{task_status_update,{{1312,529585,68188},0}}]}, {trap_exit,true}, {status,running}, {heap_size,4181}, {stack_size,24}, {reductions,38989}], [{neighbour, [{pid,<0.1031.0>}, {registered_name,[]}, {initial_call, {couch_event_sup,init,['Argument__1']}}, {current_function,{gen_server,loop,6}}, {ancestors, = [<0.1009.0>,couch_rep_sup,couch_primary_services, couch_server_sup,<0.33.0>]}, {messages,[{'$gen_cast',stop}]}, {links,[<0.1009.0>,<0.86.0>]}, {dictionary,[]}, {trap_exit,false}, {status,runnable}, {heap_size,233}, {stack_size,9}, {reductions,36}]}]]}} [Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1026.0>] ** Generic server = <0.1026.0> terminating=20 ** Last message in was {'EXIT',<0.1009.0>, {http_request_failed, <<"failed to replicate = http://137.mmm.mmm.mmm:5984/udc/">>}} ** When Server state =3D=3D {state,<0.1027.0>, {<0.1027.0>,#Ref<0.0.0.32151>}, <0.1009.0>,false,1001,nil, = {[{<<"e342b02dabe55a4c06c546ad01e24c90564e3813">>, 5005, [{2, [many docs from debug output...] And finally : * Reason for termination =3D=3D=20 ** {http_request_failed,<<"failed to replicate = http://137.mmm.mmm.mmm:5984/udc/">>} The target Server log entry shows 10 times (or so): [Fri, 05 Aug 2011 08:09:15 GMT] [debug] [<0.4821.0>] 'POST' = /udc/_bulk_docs {1,1} from "130.mmm.mmm.mmm" Headers: [{'Accept',"application/json"}, {'Accept-Encoding',"gzip"}, {'Content-Length',"43227543"}, {'Content-Type',"application/json"}, {'Host',"137.mmm.mmm.mmm:5984"}, {'User-Agent',"CouchDB/1.1.0"}, {"X-Couch-Full-Commit","false"}] [Fri, 05 Aug 2011 08:09:15 GMT] [debug] [<0.4821.0>] OAuth Params: [] --=20 Eclipse Code Recommenders: w www.eclipse.org/recommenders tw www.twitter.com/marcelbruch --Apple-Mail-50--783384106--