CouchDB 1.1 _replicator replications do not retry on timeout ------------------------------------------------------------ Key: COUCHDB-1194 URL: https://issues.apache.org/jira/browse/COUCHDB-1194 Project: CouchDB Issue Type: Bug Components: Replication Affects Versions: 1.1 Environment: Linux/amd64, Erlang 13B4 Reporter: Dirkjan Ochtman With 1.1, we started using the _replicator database to try and ensure that two continuous replication processes always run. To help with this, we have max_replication_retry_count = 100 in the replicator section of the configuration. However, it seems that, when there's a "changes loop timeout", the retry process isn't used, so that I end up without active replication (and the replication state in the documents gets set to error). This seems like a bug. [Wed, 15 Jun 2011 05:52:33 GMT] [debug] [<0.159.0>] target doesn't need a full commit [Wed, 15 Jun 2011 05:52:33 GMT] [info] [<0.159.0>] recording a checkpoint for http://10.8.0.1:5984/fix/ -> fix at source update_seq 1340615 [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.167.0>] changes loop timeout, no data received from http://10.8.0.1:5984/fix/ [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.167.0>] ** Generic server <0.167.0> terminating ** Last message in was changes_timeout ** When Server state == {state,nil,nil, [<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[], "_changes", [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}], [{"style",all_docs}, {"heartbeat",10000}, {"since",1340538}, {"feed",continuous}], get,nil, [{stream_to,{<0.167.0>,once}}, {response_format,binary}], 10,500,<0.168.0>}, 1340538, [{<<"_id">>, <<"c7311d13160739f3d1b9458b2300242e">>}, {<<"_rev">>, <<"18-94d8df687ec8434de355fa1c630fb14d">>}, {<<"source">>,<<"http://10.8.0.1:5984/fix">>}, {<<"target">>,<<"fix">>}, {<<"continuous">>,true}, {<<"_replication_state_time">>, <<"2011-06-15T07:23:15+02:00">>}, {<<"_replication_id">>, <<"812005cfdfded6c80a444f85d71caffd">>}, {<<"_replication_state">>,<<"triggered">>}]], 1340538,<0.168.0>, {1308,115420,466421}, false,0,<<>>, {<0.170.0>,#Ref<0.0.3.202756>}, {[],[]}, nil} ** Reason for termination == ** changes_timeout [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.167.0>] {error_report,<0.32.0>, {<0.167.0>,crash_report, [[{initial_call, {couch_rep_changes_feed,init,['Argument__1']}}, {pid,<0.167.0>}, {registered_name,[]}, {error_info, {exit,changes_timeout, [{gen_server,terminate,6}, {proc_lib,init_p_do_apply,3}]}}, {ancestors, [<0.159.0>,couch_rep_sup, couch_primary_services,couch_server_sup, <0.33.0>]}, {messages, [{ibrowse_async_response, {1308,115420,466421}, {error,closing_on_request}}]}, {links,[<0.159.0>]}, {dictionary, [{timeout, {1308117189023695,#Ref<0.0.3.205192>}}]}, {trap_exit,true}, {status,running}, {heap_size,1597}, {stack_size,24}, {reductions,39675}], []]}} [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.169.0>] ** Generic server <0.169.0> terminating ** Last message in was {'EXIT',<0.170.0>, {changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}} ** When Server state == {state,nil,nil,<0.159.0>,false,0, {<0.172.0>,#Ref<0.0.3.202760>}, {[],[]}, 1340615,1340615,1340615} ** Reason for termination == ** {changes_timeout,{gen_server,call,[<0.167.0>,next_changes,infinity]}} [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.169.0>] {error_report,<0.32.0>, {<0.169.0>,crash_report, [[{initial_call, {couch_rep_missing_revs,init,['Argument__1']}}, {pid,<0.169.0>}, {registered_name,[]}, {error_info, {exit, {changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}, [{gen_server,terminate,6}, {proc_lib,init_p_do_apply,3}]}}, {ancestors, [<0.159.0>,couch_rep_sup,couch_primary_services, couch_server_sup,<0.33.0>]}, {messages,[]}, {links,[<0.159.0>]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,2584}, {stack_size,24}, {reductions,44057}], []]}} [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.171.0>] ** Generic server <0.171.0> terminating ** Last message in was {'EXIT',<0.172.0>, {{changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}, {gen_server,call, [<0.169.0>,next_missing_revs,infinity]}}} ** When Server state == {state,<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, <0.169.0>,<0.172.0>,[],0, {[],[]}, {<0.173.0>,#Ref<0.0.3.202798>}, false,0,nil,[], [1340615]} ** Reason for termination == ** {function_clause, [{couch_rep_reader,handle_info, [{'EXIT',<0.172.0>, {{changes_timeout, {gen_server,call,[<0.167.0>,next_changes,infinity]}}, {gen_server,call,[<0.169.0>,next_missing_revs,infinity]}}}, {state,<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary},{inactivity_timeout,30000}], 10,500,nil}, <0.169.0>,<0.172.0>,[],0, {[],[]}, {<0.173.0>,#Ref<0.0.3.202798>}, false,0,nil,[], [1340615]}]}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.171.0>] {error_report,<0.32.0>, {<0.171.0>,crash_report, [[{initial_call,{couch_rep_reader,init,['Argument__1']}}, {pid,<0.171.0>}, {registered_name,[]}, {error_info, {exit, {function_clause, [{couch_rep_reader,handle_info, [{'EXIT',<0.172.0>, {{changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}, {gen_server,call, [<0.169.0>,next_missing_revs,infinity]}}}, {state,<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, <0.169.0>,<0.172.0>,[],0, {[],[]}, {<0.173.0>,#Ref<0.0.3.202798>}, false,0,nil,[], [1340615]}]}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}, [{gen_server,terminate,6}, {proc_lib,init_p_do_apply,3}]}}, {ancestors, [<0.159.0>,couch_rep_sup,couch_primary_services, couch_server_sup,<0.33.0>]}, {messages,[]}, {links,[<0.159.0>]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,987}, {stack_size,24}, {reductions,4847}], []]}} [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.94.0>] DDocProc found for DDocKey: {<<"_design/_replicator">>, <<"1-5bfa2c99eefe2b2eb4962db50aa3cfd4">>} [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.484.0>] OS Process #Port<0.2288> Input :: ["reset",{"reduce_limit":false,"timeout":5000}] [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.484.0>] OS Process #Port<0.2288> Output :: true [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.484.0>] OS Process #Port<0.2288> Input :: ["ddoc","_design/_replicator",["validate_doc_update"],[{"_id":"c7311d13160739f3d$ [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.484.0>] OS Process #Port<0.2288> Output :: 1 [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.159.0>] ** Generic server <0.159.0> terminating ** Last message in was {'EXIT',<0.167.0>,changes_timeout} ** When Server state == {state,<0.167.0>,<0.169.0>,<0.171.0>,<0.173.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, {db,<0.163.0>,<0.164.0>,nil, <<"1308115420372234">>,<0.161.0>,<0.165.0>, {db_header,5,1340629,0, {4370299631,{1340449,106}}, {4370302232,1340555}, {4370327477,[]}, 0,nil,nil,1000}, 1340629, {btree,<0.161.0>, {4370299631,{1340449,106}}, #Fun, #Fun, #Fun, #Fun}, {btree,<0.161.0>, {4370302232,1340555}, #Fun, #Fun, #Fun, #Fun}, {btree,<0.161.0>, {4370327477,[]}, #Fun, #Fun, #Fun,nil}, 1340629,<<"fix">>, "/var/lib/couchdb/fix.couch",[],[],nil, {user_ctx,null,[],undefined}, nil,1000, [before_header,after_header,on_file_open], false}, true,false, ["812005cfdfded6c80a444f85d71caffd", {[{<<"_id">>, <<"c7311d13160739f3d1b9458b2300242e">>}, {<<"_rev">>, <<"18-94d8df687ec8434de355fa1c630fb14d">>}, {<<"source">>,<<"http://10.8.0.1:5984/fix">>}, {<<"target">>,<<"fix">>}, {<<"continuous">>,true}, {<<"_replication_state_time">>, <<"2011-06-15T07:23:15+02:00">>}, {<<"_replication_id">>, <<"812005cfdfded6c80a444f85d71caffd">>}, {<<"_replication_state">>,<<"triggered">>}]}, {user_ctx,null,[],undefined}], nil,1340538, [{[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"start_time">>, <<"Wed, 15 Jun 2011 05:23:15 GMT">>}, {<<"end_time">>, <<"Wed, 15 Jun 2011 05:23:27 GMT">>}, {<<"start_last_seq">>,1339025}, {<<"end_last_seq">>,1340538}, {<<"recorded_seq">>,1340538}, {<<"missing_checked">>,0}, {<<"missing_found">>,1513}, {<<"docs_read">>,1513}, {<<"docs_written">>,1513}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"bfadb2b60b340623b323ea6a5070179b">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 15:18:24 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 16:50:33 GMT">>}, {<<"start_last_seq">>,1338839}, {<<"end_last_seq">>,1339025}, {<<"recorded_seq">>,1339025}, {<<"missing_checked">>,0}, {<<"missing_found">>,186}, {<<"docs_read">>,186}, {<<"docs_written">>,186}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"08e778bf393baa450b1912d6db9c7d9a">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 06:40:12 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 15:17:32 GMT">>}, {<<"start_last_seq">>,1337663}, {<<"end_last_seq">>,1338839}, {<<"recorded_seq">>,1338839}, {<<"missing_checked">>,0}, {<<"missing_found">>,1176}, {<<"docs_read">>,1176}, {<<"docs_written">>,1176}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"2fd14a14a4aa64f082bc3ac20255a426">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 05:35:00 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 06:05:48 GMT">>}, {<<"start_last_seq">>,1335813}, {<<"end_last_seq">>,1337663}, {<<"recorded_seq">>,1337663}, {<<"missing_checked">>,0}, {<<"missing_found">>,1850}, {<<"docs_read">>,1850}, {<<"docs_written">>,1850}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"57a21ecbb6fbb3531102f91614d9111b">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:44 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 14:49:14 GMT">>}, {<<"start_last_seq">>,1334997}, {<<"end_last_seq">>,1335813}, {<<"recorded_seq">>,1335813}, {<<"missing_checked">>,0}, {<<"missing_found">>,816}, {<<"docs_read">>,816}, {<<"docs_written">>,816}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"23fe3ab38f84393cc5994ee139ebe4ed">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:33 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 08:17:38 GMT">>}, {<<"start_last_seq">>,1328083}, {<<"end_last_seq">>,1334997}, {<<"recorded_seq">>,1334997}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"0263852cb022ff6a08648973879d805b">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:34:20 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 15:44:26 GMT">>}, {<<"start_last_seq">>,1327001}, {<<"end_last_seq">>,1328083}, {<<"recorded_seq">>,1328083}, {<<"missing_checked">>,0}, {<<"missing_found">>,1081}, {<<"docs_read">>,1081}, {<<"docs_written">>,1081}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"c1dd0eb18f98ffd847db77724a4e4e84">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:23:59 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:34:13 GMT">>}, {<<"start_last_seq">>,1326965}, {<<"end_last_seq">>,1327001}, {<<"recorded_seq">>,1327001}, {<<"missing_checked">>,0}, {<<"missing_found">>,33}, {<<"docs_read">>,32}, {<<"docs_written">>,32}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"92eaa73aaea009754c5c46a53315c541">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:17:30 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:23:34 GMT">>}, {<<"start_last_seq">>,1326801}, {<<"end_last_seq">>,1326965}, {<<"recorded_seq">>,1326965}, {<<"missing_checked">>,0}, {<<"missing_found">>,164}, {<<"docs_read">>,164}, {<<"docs_written">>,164}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"88476262845a78fab68493b83ece768a">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:55:53 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 06:10:49 GMT">>}, {<<"start_last_seq">>,1325687}, {<<"end_last_seq">>,1326801}, {<<"recorded_seq">>,1326801}, {<<"missing_checked">>,0}, {<<"missing_found">>,1114}, {<<"docs_read">>,1114}, {<<"docs_written">>,1114}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"a8ebd3ff85e1d741de4e1c29e16c62c7">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:47:15 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:55:25 GMT">>}, {<<"start_last_seq">>,31502}, {<<"end_last_seq">>,1325687}, {<<"recorded_seq">>,1325687}, {<<"missing_checked">>,0}, {<<"missing_found">>,20}, {<<"docs_read">>,20}, {<<"docs_written">>,20}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"fdaec0ed212acb470aebd33a092999c2">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:46:38 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:47:02 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,31502}, {<<"recorded_seq">>,31502}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}], <<"a6e39084d0605c4719d7794cf2713989">>, {doc, <<"_local/812005cfdfded6c80a444f85d71caffd">>, {0,[<<"2312">>]}, {[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"source_last_seq">>,1340538}, {<<"replication_id_version">>,2}, {<<"history">>, [{[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"start_time">>, <<"Wed, 15 Jun 2011 05:23:15 GMT">>}, {<<"end_time">>, <<"Wed, 15 Jun 2011 05:23:27 GMT">>}, {<<"start_last_seq">>,1339025}, {<<"end_last_seq">>,1340538}, {<<"recorded_seq">>,1340538}, {<<"missing_checked">>,0}, {<<"missing_found">>,1513}, {<<"docs_read">>,1513}, {<<"docs_written">>,1513}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"bfadb2b60b340623b323ea6a5070179b">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 15:18:24 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 16:50:33 GMT">>}, {<<"start_last_seq">>,1338839}, {<<"end_last_seq">>,1339025}, {<<"recorded_seq">>,1339025}, {<<"missing_checked">>,0}, {<<"missing_found">>,186}, {<<"docs_read">>,186}, {<<"docs_written">>,186}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"08e778bf393baa450b1912d6db9c7d9a">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 06:40:12 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 15:17:32 GMT">>}, {<<"start_last_seq">>,1337663}, {<<"end_last_seq">>,1338839}, {<<"recorded_seq">>,1338839}, {<<"missing_checked">>,0}, {<<"missing_found">>,1176}, {<<"docs_read">>,1176}, {<<"docs_written">>,1176}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"2fd14a14a4aa64f082bc3ac20255a426">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 05:35:00 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 06:05:48 GMT">>}, {<<"start_last_seq">>,1335813}, {<<"end_last_seq">>,1337663}, {<<"recorded_seq">>,1337663}, {<<"missing_checked">>,0}, {<<"missing_found">>,1850}, {<<"docs_read">>,1850}, {<<"docs_written">>,1850}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"57a21ecbb6fbb3531102f91614d9111b">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:44 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 14:49:14 GMT">>}, {<<"start_last_seq">>,1334997}, {<<"end_last_seq">>,1335813}, {<<"recorded_seq">>,1335813}, {<<"missing_checked">>,0}, {<<"missing_found">>,816}, {<<"docs_read">>,816}, {<<"docs_written">>,816}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"23fe3ab38f84393cc5994ee139ebe4ed">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:33 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 08:17:38 GMT">>}, {<<"start_last_seq">>,1328083}, {<<"end_last_seq">>,1334997}, {<<"recorded_seq">>,1334997}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"0263852cb022ff6a08648973879d805b">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:34:20 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 15:44:26 GMT">>}, {<<"start_last_seq">>,1327001}, {<<"end_last_seq">>,1328083}, {<<"recorded_seq">>,1328083}, {<<"missing_checked">>,0}, {<<"missing_found">>,1081}, {<<"docs_read">>,1081}, {<<"docs_written">>,1081}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"c1dd0eb18f98ffd847db77724a4e4e84">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:23:59 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:34:13 GMT">>}, {<<"start_last_seq">>,1326965}, {<<"end_last_seq">>,1327001}, {<<"recorded_seq">>,1327001}, {<<"missing_checked">>,0}, {<<"missing_found">>,33}, {<<"docs_read">>,32}, {<<"docs_written">>,32}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"92eaa73aaea009754c5c46a53315c541">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:17:30 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:23:34 GMT">>}, {<<"start_last_seq">>,1326801}, {<<"end_last_seq">>,1326965}, {<<"recorded_seq">>,1326965}, {<<"missing_checked">>,0}, {<<"missing_found">>,164}, {<<"docs_read">>,164}, {<<"docs_written">>,164}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"88476262845a78fab68493b83ece768a">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:55:53 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 06:10:49 GMT">>}, {<<"start_last_seq">>,1325687}, {<<"end_last_seq">>,1326801}, {<<"recorded_seq">>,1326801}, {<<"missing_checked">>,0}, {<<"missing_found">>,1114}, {<<"docs_read">>,1114}, {<<"docs_written">>,1114}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"a8ebd3ff85e1d741de4e1c29e16c62c7">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:47:15 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:55:25 GMT">>}, {<<"start_last_seq">>,31502}, {<<"end_last_seq">>,1325687}, {<<"recorded_seq">>,1325687}, {<<"missing_checked">>,0}, {<<"missing_found">>,20}, {<<"docs_read">>,20}, {<<"docs_written">>,20}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"fdaec0ed212acb470aebd33a092999c2">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:46:38 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:47:02 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,31502}, {<<"recorded_seq">>,31502}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}]}]}, [],false,[]}, {doc, <<"_local/812005cfdfded6c80a444f85d71caffd">>, {0,[<<"2312">>]}, {[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"source_last_seq">>,1340538}, {<<"replication_id_version">>,2}, {<<"history">>, [{[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"start_time">>, <<"Wed, 15 Jun 2011 05:23:15 GMT">>}, {<<"end_time">>, <<"Wed, 15 Jun 2011 05:23:27 GMT">>}, {<<"start_last_seq">>,1339025}, {<<"end_last_seq">>,1340538}, {<<"recorded_seq">>,1340538}, {<<"missing_checked">>,0}, {<<"missing_found">>,1513}, {<<"docs_read">>,1513}, {<<"docs_written">>,1513}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"bfadb2b60b340623b323ea6a5070179b">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 15:18:24 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 16:50:33 GMT">>}, {<<"start_last_seq">>,1338839}, {<<"end_last_seq">>,1339025}, {<<"recorded_seq">>,1339025}, {<<"missing_checked">>,0}, {<<"missing_found">>,186}, {<<"docs_read">>,186}, {<<"docs_written">>,186}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"08e778bf393baa450b1912d6db9c7d9a">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 06:40:12 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 15:17:32 GMT">>}, {<<"start_last_seq">>,1337663}, {<<"end_last_seq">>,1338839}, {<<"recorded_seq">>,1338839}, {<<"missing_checked">>,0}, {<<"missing_found">>,1176}, {<<"docs_read">>,1176}, {<<"docs_written">>,1176}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"2fd14a14a4aa64f082bc3ac20255a426">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 05:35:00 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 06:05:48 GMT">>}, {<<"start_last_seq">>,1335813}, {<<"end_last_seq">>,1337663}, {<<"recorded_seq">>,1337663}, {<<"missing_checked">>,0}, {<<"missing_found">>,1850}, {<<"docs_read">>,1850}, {<<"docs_written">>,1850}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"57a21ecbb6fbb3531102f91614d9111b">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:44 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 14:49:14 GMT">>}, {<<"start_last_seq">>,1334997}, {<<"end_last_seq">>,1335813}, {<<"recorded_seq">>,1335813}, {<<"missing_checked">>,0}, {<<"missing_found">>,816}, {<<"docs_read">>,816}, {<<"docs_written">>,816}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"23fe3ab38f84393cc5994ee139ebe4ed">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:33 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 08:17:38 GMT">>}, {<<"start_last_seq">>,1328083}, {<<"end_last_seq">>,1334997}, {<<"recorded_seq">>,1334997}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"0263852cb022ff6a08648973879d805b">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:34:20 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 15:44:26 GMT">>}, {<<"start_last_seq">>,1327001}, {<<"end_last_seq">>,1328083}, {<<"recorded_seq">>,1328083}, {<<"missing_checked">>,0}, {<<"missing_found">>,1081}, {<<"docs_read">>,1081}, {<<"docs_written">>,1081}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"c1dd0eb18f98ffd847db77724a4e4e84">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:23:59 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:34:13 GMT">>}, {<<"start_last_seq">>,1326965}, {<<"end_last_seq">>,1327001}, {<<"recorded_seq">>,1327001}, {<<"missing_checked">>,0}, {<<"missing_found">>,33}, {<<"docs_read">>,32}, {<<"docs_written">>,32}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"92eaa73aaea009754c5c46a53315c541">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:17:30 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:23:34 GMT">>}, {<<"start_last_seq">>,1326801}, {<<"end_last_seq">>,1326965}, {<<"recorded_seq">>,1326965}, {<<"missing_checked">>,0}, {<<"missing_found">>,164}, {<<"docs_read">>,164}, {<<"docs_written">>,164}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"88476262845a78fab68493b83ece768a">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:55:53 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 06:10:49 GMT">>}, {<<"start_last_seq">>,1325687}, {<<"end_last_seq">>,1326801}, {<<"recorded_seq">>,1326801}, {<<"missing_checked">>,0}, {<<"missing_found">>,1114}, {<<"docs_read">>,1114}, {<<"docs_written">>,1114}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"a8ebd3ff85e1d741de4e1c29e16c62c7">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:47:15 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:55:25 GMT">>}, {<<"start_last_seq">>,31502}, {<<"end_last_seq">>,1325687}, {<<"recorded_seq">>,1325687}, {<<"missing_checked">>,0}, {<<"missing_found">>,20}, {<<"docs_read">>,20}, {<<"docs_written">>,20}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"fdaec0ed212acb470aebd33a092999c2">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:46:38 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:47:02 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,31502}, {<<"recorded_seq">>,31502}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}]}]}, [],false,[]}, "Wed, 15 Jun 2011 05:23:40 GMT", <<"1307626215287183">>,<<"1308115420372234">>, {[{<<"session_id">>, <<"a6e39084d0605c4719d7794cf2713989">>}, {<<"source_last_seq">>,1340615}, {<<"replication_id_version">>,2}, {<<"history">>, [{[{<<"session_id">>, <<"a6e39084d0605c4719d7794cf2713989">>}, {<<"start_time">>, <<"Wed, 15 Jun 2011 05:23:40 GMT">>}, {<<"end_time">>, <<"Wed, 15 Jun 2011 05:52:33 GMT">>}, {<<"start_last_seq">>,1340538}, {<<"end_last_seq">>,1340615}, {<<"recorded_seq">>,1340615}, {<<"missing_checked">>,0}, {<<"missing_found">>,77}, {<<"docs_read">>,77}, {<<"docs_written">>,77}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"c32823f2d4efa8c5ab5e3dfb10a05fda">>}, {<<"start_time">>, <<"Wed, 15 Jun 2011 05:23:15 GMT">>}, {<<"end_time">>, <<"Wed, 15 Jun 2011 05:23:27 GMT">>}, {<<"start_last_seq">>,1339025}, {<<"end_last_seq">>,1340538}, {<<"recorded_seq">>,1340538}, {<<"missing_checked">>,0}, {<<"missing_found">>,1513}, {<<"docs_read">>,1513}, {<<"docs_written">>,1513}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"bfadb2b60b340623b323ea6a5070179b">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 15:18:24 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 16:50:33 GMT">>}, {<<"start_last_seq">>,1338839}, {<<"end_last_seq">>,1339025}, {<<"recorded_seq">>,1339025}, {<<"missing_checked">>,0}, {<<"missing_found">>,186}, {<<"docs_read">>,186}, {<<"docs_written">>,186}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"08e778bf393baa450b1912d6db9c7d9a">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 06:40:12 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 15:17:32 GMT">>}, {<<"start_last_seq">>,1337663}, {<<"end_last_seq">>,1338839}, {<<"recorded_seq">>,1338839}, {<<"missing_checked">>,0}, {<<"missing_found">>,1176}, {<<"docs_read">>,1176}, {<<"docs_written">>,1176}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"2fd14a14a4aa64f082bc3ac20255a426">>}, {<<"start_time">>, <<"Tue, 14 Jun 2011 05:35:00 GMT">>}, {<<"end_time">>, <<"Tue, 14 Jun 2011 06:05:48 GMT">>}, {<<"start_last_seq">>,1335813}, {<<"end_last_seq">>,1337663}, {<<"recorded_seq">>,1337663}, {<<"missing_checked">>,0}, {<<"missing_found">>,1850}, {<<"docs_read">>,1850}, {<<"docs_written">>,1850}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"57a21ecbb6fbb3531102f91614d9111b">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:44 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 14:49:14 GMT">>}, {<<"start_last_seq">>,1334997}, {<<"end_last_seq">>,1335813}, {<<"recorded_seq">>,1335813}, {<<"missing_checked">>,0}, {<<"missing_found">>,816}, {<<"docs_read">>,816}, {<<"docs_written">>,816}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"23fe3ab38f84393cc5994ee139ebe4ed">>}, {<<"start_time">>, <<"Mon, 13 Jun 2011 08:17:33 GMT">>}, {<<"end_time">>, <<"Mon, 13 Jun 2011 08:17:38 GMT">>}, {<<"start_last_seq">>,1328083}, {<<"end_last_seq">>,1334997}, {<<"recorded_seq">>,1334997}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"0263852cb022ff6a08648973879d805b">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:34:20 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 15:44:26 GMT">>}, {<<"start_last_seq">>,1327001}, {<<"end_last_seq">>,1328083}, {<<"recorded_seq">>,1328083}, {<<"missing_checked">>,0}, {<<"missing_found">>,1081}, {<<"docs_read">>,1081}, {<<"docs_written">>,1081}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"c1dd0eb18f98ffd847db77724a4e4e84">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:23:59 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:34:13 GMT">>}, {<<"start_last_seq">>,1326965}, {<<"end_last_seq">>,1327001}, {<<"recorded_seq">>,1327001}, {<<"missing_checked">>,0}, {<<"missing_found">>,33}, {<<"docs_read">>,32}, {<<"docs_written">>,32}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"92eaa73aaea009754c5c46a53315c541">>}, {<<"start_time">>, <<"Fri, 10 Jun 2011 07:17:30 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 07:23:34 GMT">>}, {<<"start_last_seq">>,1326801}, {<<"end_last_seq">>,1326965}, {<<"recorded_seq">>,1326965}, {<<"missing_checked">>,0}, {<<"missing_found">>,164}, {<<"docs_read">>,164}, {<<"docs_written">>,164}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"88476262845a78fab68493b83ece768a">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:55:53 GMT">>}, {<<"end_time">>, <<"Fri, 10 Jun 2011 06:10:49 GMT">>}, {<<"start_last_seq">>,1325687}, {<<"end_last_seq">>,1326801}, {<<"recorded_seq">>,1326801}, {<<"missing_checked">>,0}, {<<"missing_found">>,1114}, {<<"docs_read">>,1114}, {<<"docs_written">>,1114}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"a8ebd3ff85e1d741de4e1c29e16c62c7">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:47:15 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:55:25 GMT">>}, {<<"start_last_seq">>,31502}, {<<"end_last_seq">>,1325687}, {<<"recorded_seq">>,1325687}, {<<"missing_checked">>,0}, {<<"missing_found">>,20}, {<<"docs_read">>,20}, {<<"docs_written">>,20}, {<<"doc_write_failures">>,0}]}, {[{<<"session_id">>, <<"fdaec0ed212acb470aebd33a092999c2">>}, {<<"start_time">>, <<"Thu, 09 Jun 2011 20:46:38 GMT">>}, {<<"end_time">>, <<"Thu, 09 Jun 2011 20:47:02 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,31502}, {<<"recorded_seq">>,31502}, {<<"missing_checked">>,0}, {<<"missing_found">>,0}, {<<"docs_read">>,0}, {<<"docs_written">>,0}, {<<"doc_write_failures">>,0}]}]}]}, [],false,1340615,61495, {[{<<"_id">>, <<"c7311d13160739f3d1b9458b2300242e">>}, {<<"_rev">>, <<"18-94d8df687ec8434de355fa1c630fb14d">>}, {<<"source">>,<<"http://10.8.0.1:5984/fix">>}, {<<"target">>,<<"fix">>}, {<<"continuous">>,true}, {<<"_replication_state_time">>, <<"2011-06-15T07:23:15+02:00">>}, {<<"_replication_id">>, <<"812005cfdfded6c80a444f85d71caffd">>}, {<<"_replication_state">>,<<"triggered">>}]}, nil,<0.174.0>} ** Reason for termination == ** changes_timeout [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.159.0>] {error_report,<0.32.0>, {<0.159.0>,crash_report, [[{initial_call,{couch_rep,init,['Argument__1']}}, {pid,<0.159.0>}, {registered_name,[]}, {error_info, {exit,changes_timeout, [{gen_server,terminate,6}, {proc_lib,init_p_do_apply,3}]}}, {ancestors, [couch_rep_sup,couch_primary_services, couch_server_sup,<0.33.0>]}, {messages, [{'EXIT',<0.169.0>, {changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}}, {'EXIT',<0.174.0>,normal}, {'EXIT',<0.171.0>, {function_clause, [{couch_rep_reader,handle_info, [{'EXIT',<0.172.0>, {{changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}, {gen_server,call, [<0.169.0>,next_missing_revs,infinity]}}}, {state,<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, <0.169.0>,<0.172.0>,[],0, {[],[]}, {<0.173.0>,#Ref<0.0.3.202798>}, false,0,nil,[], [1340615]}]}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}}, {'EXIT',<0.173.0>, {{function_clause, [{couch_rep_reader,handle_info, [{'EXIT',<0.172.0>, {{changes_timeout, {gen_server,call, [<0.167.0>,next_changes,infinity]}}, {gen_server,call, [<0.169.0>,next_missing_revs,infinity]}}}, {state,<0.159.0>, {http_db,"http://10.8.0.1:5984/fix/",[],[], [{"User-Agent","CouchDB/1.1.0"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, <0.169.0>,<0.172.0>,[],0, {[],[]}, {<0.173.0>,#Ref<0.0.3.202798>}, false,0,nil,[], [1340615]}]}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}, {gen_server,call, [<0.171.0>,next_docs,infinity]}}}]}, {links,[<0.83.0>]}, {dictionary, [{task_status_update,{{1308,117148,88291},0}}]}, {trap_exit,true}, {status,running}, {heap_size,6765}, {stack_size,24}, {reductions,271204}], []]}} [Wed, 15 Jun 2011 05:53:09 GMT] [error] [<0.83.0>] {error_report,<0.32.0>, {<0.83.0>,supervisor_report, [{supervisor,{local,couch_rep_sup}}, {errorContext,child_terminated}, {reason,changes_timeout}, {offender, [{pid,<0.159.0>}, {name, "812005cfdfded6c80a444f85d71caffd+continuous"}, {mfa, {gen_server,start_link, [couch_rep, ["812005cfdfded6c80a444f85d71caffd", {[{<<"_id">>, <<"c7311d13160739f3d1b9458b2300242e">>}, {<<"_rev">>, <<"18-94d8df687ec8434de355fa1c630fb14d">>}, {<<"source">>, <<"http://10.8.0.1:5984/fix">>}, {<<"target">>,<<"fix">>}, {<<"continuous">>,true}, {<<"_replication_state_time">>, <<"2011-06-15T07:23:15+02:00">>}, {<<"_replication_id">>, <<"812005cfdfded6c80a444f85d71caffd">>}, {<<"_replication_state">>, <<"triggered">>}]}, {user_ctx,null,[],undefined}], []]}}, {restart_type,temporary}, {shutdown,1}, {child_type,worker}]}]}} [Wed, 15 Jun 2011 05:53:09 GMT] [debug] [<0.134.0>] Include Doc: <<"c7311d13160739f3d1b9458b2300242e">> {19, <<220,6, 99,17, 13,51, 140,159, 244,19, 55,198, 109,204, 109,149>>} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira