Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 34597 invoked from network); 4 Nov 2009 13:43:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Nov 2009 13:43:10 -0000 Received: (qmail 30057 invoked by uid 500); 4 Nov 2009 13:43:08 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 29958 invoked by uid 500); 4 Nov 2009 13:43:08 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 29948 invoked by uid 99); 4 Nov 2009 13:43:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 13:43:07 +0000 X-ASF-Spam-Status: No, hits=2.7 required=10.0 tests=FS_REPLICA,HS_INDEX_PARAM,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of adam.kocoloski@gmail.com designates 74.125.92.27 as permitted sender) Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 13:42:56 +0000 Received: by qw-out-2122.google.com with SMTP id 9so1310836qwb.29 for ; Wed, 04 Nov 2009 05:42:34 -0800 (PST) Received: by 10.224.0.157 with SMTP id 29mr734862qab.373.1257342154228; Wed, 04 Nov 2009 05:42:34 -0800 (PST) Received: from ?10.0.1.9? (c-71-232-49-44.hsd1.ma.comcast.net [71.232.49.44]) by mx.google.com with ESMTPS id 20sm734857qyk.1.2009.11.04.05.42.32 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 04 Nov 2009 05:42:32 -0800 (PST) Content-Type: text/plain; charset=iso-8859-1; format=flowed; delsp=yes Mime-Version: 1.0 (Apple Message framework v1076) Subject: Re: Continous Replication hangs with "start" From: Adam Kocoloski In-Reply-To: <1257333310.26854.47.camel@redemption> Date: Wed, 4 Nov 2009 08:42:31 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1257333310.26854.47.camel@redemption> To: user@couchdb.apache.org X-Mailer: Apple Mail (2.1076) X-Virus-Checked: Checked by ClamAV on apache.org Hi Simon, looks like replication is failing on the following URL: = http://10.1.1.31:5984/gangstercluster_1/ping_http:%2F%2Fredemption.intrane= t.struktur.de:2425?open_revs=3D[\=20 "26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue It gets back a response that looks like HTTP/1.1 500 Internal Server Error Server: CouchDB/0.10.0 (Erlang OTP/R13B) Date: Wed, 04 Nov 2009 11:10:19 GMT Content-Type: application/json Content-Length: 114 Cache-Control: must-revalidate {\"error\":\"json_encode\",\"reason\":\"{bad_term,=20 {26621,<<47,146,160,252,77,28,71,180,40,233,173,84,97,192,173,47>>}}\"}=20= \n" You could try the URL for yourself and confirm this to be the case. =20 The problem is probably that revision 26621-.... of this document is =20 missing. 0.10.0 had a bug where if you explicitly requested a missing =20= revision, you'd get a 500 Internal Server Response. The replicator =20 isn't in the habit of making these kinds of requests, but it's =20 probably some interaction with the full disk problem. I fixed this bug in r829919 (trunk) or r829924 (0.10.x branch, didn't =20= make 0.10.0), so if you upgraded it would likely allow replication to =20= proceed. Otherwise, I think your recourse is to make some update to =20 this document so that the replicator doesn't try to request this =20 revision anymore. Hope it helps, Adam On Nov 4, 2009, at 6:15 AM, Simon Eisenmann wrote: > Hi, > > i have been running continuous replication for the past three weeks on > three nodes each replicating to each other with continuous changes on > every node. All running fine until one node crashed cause of a full > disk. > > I emptied disk space and restarted this node. This replication never > seems to start again. Status in futon keep saying "Starting". > > I tried to restart all the nodes with no luck, all having the same > issues. > > See the errors in the couchdb.log below. > > I am running CouchDB 0.10.0 on Erlang R13B. > > I hope somebody has some sort of idea what is going on. > > > Thanks > Simon > > > [Wed, 04 Nov 2009 11:10:19 GMT] [error] [<0.2114.0>] ** Generic server > <0.2114.0> terminating > ** Last message in was {tcp,#Port<0.2219>, > <<"HTTP/1.1 500 Internal Server Error\r > \nServer: CouchDB/0.10.0 (Erlang OTP/R13B)\r\nDate: Wed, 04 Nov 2009 > 11:10:19 GMT\r\nContent-Type: application/json\r\nContent-Length: =20 > 114\r > \nCache-Control: must-revalidate\r\n\r\n{\"error\":\"json_encode\", > \"reason\": > \"{bad_term,=20 > {26621,<<47,146,160,252,77,28,71,180,40,233,173,84,97,192,173,47>>}}=20= > \"}\n">>} > ** When Server state =3D=3D > {state,"10.1.1.31",5984,false,undefined,[],false, > #Port<0.2219>, > {[{request, > {url, > > "http://10.1.1.31:5984/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > > "10.1.1.31",5984,undefined,undefined, > "/gangstercluster_1/ping_http:=20 > %2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > http}, > get, > [{response_format,binary}, > {inactivity_timeout,30000}], > {<0.2084.0>,#Ref<0.0.3.30137>}, > undefined,false, > {1257,333019,329755}, > > 1048576,false,undefined,undefined,binary}], > []}, > {request, > {url, > > "http://10.1.1.31:5984/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > > "10.1.1.31",5984,undefined,undefined, > "/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > http}, > get, > [{response_format,binary}, > {inactivity_timeout,30000}], > {<0.2084.0>,#Ref<0.0.3.30137>}, > undefined,false, > {1257,333019,329755}, > > 1048576,false,undefined,undefined,binary}, > get_body,"200",<<"[">>,0,0, > [{"Transfer-Encoding","chunked"}, > {"Server","CouchDB/0.10.0 (Erlang > OTP/R13B)"}, > {"Date","Wed, 04 Nov 2009 11:10:19 GMT"}, > {"Content-Type","application/json"}, > {"Cache-Control","must-revalidate"}], > false,undefined,undefined,true,chunked, > chunk_start,<<>>,0,262182,1,undefined} > ** Reason for termination =3D=3D > ** {function_clause,[{ibrowse_http_client,to_ascii,"r"}, > {ibrowse_http_client,hexlist_to_integer,3}, > {ibrowse_http_client,parse_11_response,2}, > {ibrowse_http_client,handle_sock_data,2}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]} > > > [Wed, 04 Nov 2009 11:10:19 GMT] [error] [<0.2114.0>] > {error_report,<0.24.0>, > {<0.2114.0>,crash_report, > [[{initial_call,{ibrowse_http_client,init,['Argument__1']}}, > {pid,<0.2114.0>}, > {registered_name,[]}, > {error_info, > {exit, > {function_clause, > [{ibrowse_http_client,to_ascii,"r"}, > {ibrowse_http_client,hexlist_to_integer,3}, > {ibrowse_http_client,parse_11_response,2}, > {ibrowse_http_client,handle_sock_data,2}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]}, > > [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, > {ancestors,[<0.139.0>,ibrowse,<0.1.0>]}, > {messages,[]}, > {links,[<0.139.0>]}, > {dictionary, > [{my_trace_flag,false}, > {ibrowse_trace_token,["10.1.1.31",58,"5984"]}, > {http_prot_vsn,"HTTP/1.1"}, > {conn_close,"false"}]}, > {trap_exit,false}, > {status,running}, > {heap_size,4181}, > {stack_size,24}, > {reductions,1570}], > []]}} > > > [Wed, 04 Nov 2009 11:12:25 GMT] [error] [<0.2339.0>] ** Generic server > <0.2339.0> terminating > ** Last message in was {tcp,#Port<0.2310>, > <<"HTTP/1.1 500 Internal Server Error\r > \nServer: CouchDB/0.10.0 (Erlang OTP/R13B)\r\nDate: Wed, 04 Nov 2009 > 11:12:25 GMT\r\nContent-Type: application/json\r\nContent-Length: =20 > 114\r > \nCache-Control: must-revalidate\r\n\r\n{\"error\":\"json_encode\", > \"reason\": > \"{bad_term,=20 > {26621,<<47,146,160,252,77,28,71,180,40,233,173,84,97,192,173,47>>}}=20= > \"}\n">>} > ** When Server state =3D=3D > {state,"10.1.1.31",5984,false,undefined,[],false, > #Port<0.2310>, > {[{request, > {url, > > "http://10.1.1.31:5984/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > > "10.1.1.31",5984,undefined,undefined, > "/gangstercluster_1/ping_http:=20 > %2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > http}, > get, > [{response_format,binary}, > {inactivity_timeout,30000}], > {<0.2084.0>,#Ref<0.0.3.120029>}, > undefined,false, > {1257,333145,499104}, > > 1048576,false,undefined,undefined,binary}], > []}, > {request, > {url, > > "http://10.1.1.31:5984/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > > "10.1.1.31",5984,undefined,undefined, > "/gangstercluster_1/ping_http:%2F% > 2Fredemption.intranet.struktur.de:2425?open_revs=3D=20 > [\"26621-2f92a0fc4d1c47b428e9ad5461c0ad2f\"]&revs=3Dtrue&latest=3Dtrue",= > http}, > get, > [{response_format,binary}, > {inactivity_timeout,30000}], > {<0.2084.0>,#Ref<0.0.3.120029>}, > undefined,false, > {1257,333145,499104}, > > 1048576,false,undefined,undefined,binary}, > get_body,"200",<<"[">>,0,0, > [{"Transfer-Encoding","chunked"}, > {"Server","CouchDB/0.10.0 (Erlang > OTP/R13B)"}, > {"Date","Wed, 04 Nov 2009 11:12:25 GMT"}, > {"Content-Type","application/json"}, > {"Cache-Control","must-revalidate"}], > false,undefined,undefined,true,chunked, > chunk_start,<<>>,0,311334,1,undefined} > ** Reason for termination =3D=3D > ** {function_clause,[{ibrowse_http_client,to_ascii,"r"}, > {ibrowse_http_client,hexlist_to_integer,3}, > {ibrowse_http_client,parse_11_response,2}, > {ibrowse_http_client,handle_sock_data,2}, > {gen_server,handle_msg,5}, > {proc_lib,init_p_do_apply,3}]} > > > > --=20 > Simon Eisenmann > > [ mailto:simon@struktur.de ] > > [ struktur AG | Kronenstra=DFe 22a | D-70173 Stuttgart ] > [ T. +49.711.896656.0 | F.+49.711.89665610 ] > [ http://www.struktur.de | mailto:info@struktur.de ]