Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A75110BAA for ; Mon, 31 Mar 2014 14:21:58 +0000 (UTC) Received: (qmail 34970 invoked by uid 500); 31 Mar 2014 14:21:30 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 34435 invoked by uid 500); 31 Mar 2014 14:21:25 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 33165 invoked by uid 99); 31 Mar 2014 14:21:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Mar 2014 14:21:17 +0000 Date: Mon, 31 Mar 2014 14:21:17 +0000 (UTC) From: "Yang Zhewei (JIRA)" To: dev@couchdb.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (COUCHDB-2217) Replication Crash MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Yang Zhewei created COUCHDB-2217: ------------------------------------ Summary: Replication Crash Key: COUCHDB-2217 URL: https://issues.apache.org/jira/browse/COUCHDB-2217 Project: CouchDB Issue Type: Bug Security Level: public (Regular issues) Components: HTTP Interface, Replication Reporter: Yang Zhewei I want to run a replication to copy a database on the source server to the target server. The source server is Windows 7 OS, and the target server is Linux OS. The size of database is about 14GB. Everytime, when the replication runs and complete and 8.6GB, then it will crash. And there is no log on the target server. Here is the error log of the source server: [Mon, 31 Mar 2014 06:31:11 GMT] [error] [emulator] Error in process <0.1256.0> with exit value: {{badmatch,<<16 bytes>>},[{couch_stream,foldl,6,[{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"},{line,96}]},{couch_doc,att_to_bin,1,[{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"},{line,385}]},{couch_doc... [Mon, 31 Mar 2014 06:31:11 GMT] [error] [<0.1244.0>] Worker <0.1256.0> died with reason: {{badmatch, <<27,109,21,85,240, 3,44,180,90,71, 64,92,107,215, 185,43>>}, [{couch_stream, foldl,6, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc, att_to_bin,1, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc, '-to_json_attachments/4-fun-0-', 4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2, [{file, "lists.erl"}, {line,1224}]}, {couch_doc, to_json_attachments, 4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc, doc_to_json_obj,2, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker, maybe_flush_docs, 3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker, '-local_process_batch/6-fun-1-', 3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]} [Mon, 31 Mar 2014 06:31:11 GMT] [error] [<0.1244.0>] Replication `043a8e83493ef0bc93cdd6a8129450c4+continuous` (`testdb` -> `http://107.252.142.195:5984/testdb/`) failed: {worker_died,<0.1256.0>, {{badmatch,<<27,109,21,85,240,3,44,180,90,71,64,92,107,215,185,43>>}, [{couch_stream,foldl,6, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc,att_to_bin,1, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc,'-to_json_attachments/4-fun-0-',4, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2,[{file,"lists.erl"},{line,1224}]}, {couch_doc,to_json_attachments,4, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc,doc_to_json_obj,2, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker,maybe_flush_docs,3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker,'-local_process_batch/6-fun-1-',3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]}} [Mon, 31 Mar 2014 06:31:11 GMT] [info] [<0.1257.0>] Retrying POST request to http://107.252.142.195:5984/testdb/_bulk_docs in 0.25 seconds due to error closing_on_request [Mon, 31 Mar 2014 06:31:11 GMT] [info] [<0.1255.0>] Retrying POST request to http://107.252.142.195:5984/testdb/_bulk_docs in 0.25 seconds due to error closing_on_request [Mon, 31 Mar 2014 06:31:11 GMT] [error] [<0.1244.0>] ** Generic server <0.1244.0> terminating ** Last message in was {'EXIT',<0.1256.0>, {{badmatch, <<27,109,21,85,240,3,44,180,90,71,64,92,107, 215,185,43>>}, [{couch_stream,foldl,6, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc,att_to_bin,1, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc,'-to_json_attachments/4-fun-0-',4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2,[{file,"lists.erl"},{line,1224}]}, {couch_doc,to_json_attachments,4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc,doc_to_json_obj,2, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker,maybe_flush_docs,3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker, '-local_process_batch/6-fun-1-',3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]}} ** When Server state == {rep_state, {rep, {"043a8e83493ef0bc93cdd6a8129450c4","+continuous"}, <<"testdb">>, {httpdb,"http://107.252.142.195:5984/testdb/",nil, [{"Accept","application/json"}, {"User-Agent","CouchDB/1.5.0"}], 30000, [{socket_options, [{keepalive,true},{nodelay,false}]}], 10,250,nil,20}, [{connection_timeout,30000}, {continuous,true}, {http_connections,20}, {retries,10}, {socket_options,[{keepalive,true},{nodelay,false}]}, {worker_batch_size,500}, {worker_processes,4}], {user_ctx,null, [<<"_admin">>], <<"{couch_httpd_auth, default_authentication_handler}">>}, null}, "testdb","http://107.252.142.195:5984/testdb/", {db,<0.1198.0>,<0.1199.0>,nil, <<"1396242275324299">>,<0.1202.0>,<0.1193.0>, <0.1208.0>, {db_header,6,2444056,0, {15387074144,{2444056,0,14465346577},98251982}, {15387077404,2444056,72824021}, {15397966514,[],1097}, 0,nil,nil,1000}, 2444056, {btree,<0.1193.0>, {15387074144,{2444056,0,14465346577},98251982}, #Fun, #Fun, #Fun, #Fun,snappy}, {btree,<0.1193.0>, {15387077404,2444056,72824021}, #Fun, #Fun, #Fun, #Fun,snappy}, {btree,<0.1193.0>, {15397966514,[],1097}, #Fun, #Fun, #Fun,nil,snappy}, 2444056,<<"testdb">>, "../var/lib/couchdb/testdb.couch",[],[],nil, {user_ctx,null, [<<"_admin">>], <<"{couch_httpd_auth, default_authentication_handler}">>}, nil,1000, [before_header,after_header,on_file_open], [{user_ctx, {user_ctx,null, [<<"_admin">>], <<"{couch_httpd_auth, default_authentication_handler}">>}}], snappy,nil,nil}, {httpdb,"http://107.252.142.195:5984/testdb/",nil, [{"Accept","application/json"}, {"User-Agent","CouchDB/1.5.0"}], 30000, [{socket_options, [{keepalive,true},{nodelay,false}]}], 10,250,<0.1245.0>,20}, [], {[{<<"session_id">>, <<"26b2909072b32fe6e1c50a3237588a0c">>}, {<<"source_last_seq">>,1880572}, {<<"replication_id_version">>,3}, {<<"history">>, [{[{<<"session_id">>, <<"26b2909072b32fe6e1c50a3237588a0c">>}, {<<"start_time">>, <<"Mon, 31 Mar 2014 05:05:10 GMT">>}, {<<"end_time">>, <<"Mon, 31 Mar 2014 06:31:11 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,1880572}, {<<"recorded_seq">>,1880572}, {<<"missing_checked">>,1880572}, {<<"missing_found">>,1880572}, {<<"docs_read">>,1880572}, {<<"docs_written">>,1880572}, {<<"doc_write_failures">>,0}]}]}]}, {0,0}, {3762,1880572}, {3762,1880572}, [{3763,1881072}, {3764,1881572}, {3765,1882072}, {3766,1882572}], {3762,1880572}, {doc,<<"_local/043a8e83493ef0bc93cdd6a8129450c4">>, {0,[<<"2621">>]}, {[{<<"session_id">>, <<"f65cc548315cd55fc9f92a6469fa9243">>}, {<<"source_last_seq">>,1881504}, {<<"replication_id_version">>,3}, {<<"history">>, [{[{<<"session_id">>, <<"f65cc548315cd55fc9f92a6469fa9243">>}, {<<"start_time">>, <<"Fri, 28 Mar 2014 02:44:21 GMT">>}, {<<"end_time">>, <<"Fri, 28 Mar 2014 04:12:39 GMT">>}, {<<"start_last_seq">>,0}, {<<"end_last_seq">>,1881504}, {<<"recorded_seq">>,1881504}, {<<"missing_checked">>,1881504}, {<<"missing_found">>,1881504}, {<<"docs_read">>,1881504}, {<<"docs_written">>,1881504}, {<<"doc_write_failures">>,0}]}]}]}, [],false,[]}, {doc,<<"_local/043a8e83493ef0bc93cdd6a8129450c4">>, {0,[<<"876">>]}, {[]}, [],false,[]}, "Mon, 31 Mar 2014 05:05:10 GMT", <<"1396242275324299">>,<<"1396241960004992">>,nil, <0.1251.0>,<0.1253.0>,<0.1252.0>, [<0.1255.0>,<0.1256.0>,<0.1257.0>,<0.1258.0>], {rep_stats,1880572,1880572,1880572,1880572,0}, <<"26b2909072b32fe6e1c50a3237588a0c">>,<0.1250.0>, nil,#Ref<0.0.0.13340>,nil,2444056} ** Reason for termination == ** {worker_died,<0.1256.0>, {{badmatch,<<27,109,21,85,240,3,44,180,90,71,64,92,107,215,185,43>>}, [{couch_stream,foldl,6, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc,att_to_bin,1, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc,'-to_json_attachments/4-fun-0-',4, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2,[{file,"lists.erl"},{line,1224}]}, {couch_doc,to_json_attachments,4, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc,doc_to_json_obj,2, [{file,"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker,maybe_flush_docs,3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker,'-local_process_batch/6-fun-1-',3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]}} [Mon, 31 Mar 2014 06:31:11 GMT] [error] [<0.1244.0>] {error_report,<0.35.0>, {<0.1244.0>,crash_report, [[{initial_call, {couch_replicator,init,['Argument__1']}}, {pid,<0.1244.0>}, {registered_name,[]}, {error_info, {exit, {worker_died,<0.1256.0>, {{badmatch, <<27,109,21,85,240,3,44,180,90,71,64,92,107,215, 185,43>>}, [{couch_stream,foldl,6, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc,att_to_bin,1, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc,'-to_json_attachments/4-fun-0-',4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2,[{file,"lists.erl"},{line,1224}]}, {couch_doc,to_json_attachments,4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc,doc_to_json_obj,2, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker,maybe_flush_docs,3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker, '-local_process_batch/6-fun-1-',3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]}}, [{gen_server,terminate,6, [{file,"gen_server.erl"},{line,744}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}}, {ancestors, [couch_replicator_job_sup,couch_primary_services, couch_server_sup,<0.36.0>]}, {messages,[{'EXIT',<0.1250.0>,normal}]}, {links, [<0.1255.0>,<0.1258.0>,<0.1257.0>,<0.1251.0>, <0.1253.0>,<0.1252.0>,<0.98.0>]}, {dictionary, [{task_status_props, [{checkpointed_source_seq,1880572}, {continuous,true}, {doc_id,null}, {doc_write_failures,0}, {docs_read,1880572}, {docs_written,1880572}, {missing_revisions_found,1880572}, {progress,76}, {replication_id, <<"043a8e83493ef0bc93cdd6a8129450c4+continuous">>}, {revisions_checked,1880572}, {source,<<"testdb">>}, {source_seq,2444056}, {started_on,1396242310}, {target, <<"http://107.252.142.195:5984/testdb/">>}, {type,replication}, {updated_on,1396242310}]}, {task_status_update, {{1396,247471,495171},1000000}}]}, {trap_exit,true}, {status,running}, {heap_size,17731}, {stack_size,27}, {reductions,2710649}], [{neighbour, [{pid,<0.1254.0>}, {registered_name,[]}, {initial_call, {couch_event_sup,init,['Argument__1']}}, {current_function,{gen_server,loop,6}}, {ancestors,[<0.1252.0>]}, {messages,[]}, {links,[<0.1252.0>,<0.96.0>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,376}, {stack_size,9}, {reductions,68}]}, {neighbour, [{pid,<0.1252.0>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{gen,do_call,4}}, {ancestors,[]}, {messages,[]}, {links,[<0.1244.0>,<0.1254.0>]}, {dictionary,[{last_seq,1885854}]}, {trap_exit,false}, {status,waiting}, {heap_size,4185}, {stack_size,105}, {reductions,243382506}]}, {neighbour, [{pid,<0.1253.0>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function, {couch_replicator,changes_manager_loop_open,4}}, {ancestors,[]}, {messages,[]}, {links,[<0.1244.0>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,75113}, {stack_size,8}, {reductions,3542551}]}, {neighbour, [{pid,<0.1251.0>}, {registered_name,[]}, {initial_call, {couch_work_queue,init,['Argument__1']}}, {current_function,{gen_server,loop,6}}, {ancestors, [<0.1244.0>,couch_replicator_job_sup, couch_primary_services,couch_server_sup, <0.36.0>]}, {messages,[]}, {links,[<0.1244.0>]}, {dictionary,[]}, {trap_exit,false}, {status,waiting}, {heap_size,318187}, {stack_size,9}, {reductions,26151223}]}, {neighbour, [{pid,<0.1257.0>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{gen,do_call,4}}, {ancestors,[]}, {messages,[]}, {links,[<0.1244.0>]}, {dictionary, [{last_stats_report,{1396,247464,835830}}]}, {trap_exit,false}, {status,waiting}, {heap_size,2586}, {stack_size,41}, {reductions,1911109089}]}, {neighbour, [{pid,<0.1258.0>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{gen,do_call,4}}, {ancestors,[]}, {messages, [{#Ref<0.0.66.214409>,{error,closing_on_request}}, {'DOWN',#Ref<0.0.66.214409>,process,<0.9056.0>, normal}]}, {links,[<0.1244.0>]}, {dictionary, [{last_stats_report,{1396,247466,769580}}]}, {trap_exit,false}, {status,runnable}, {heap_size,6772}, {stack_size,59}, {reductions,1897456637}]}, {neighbour, [{pid,<0.1255.0>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{gen,do_call,4}}, {ancestors,[]}, {messages,[]}, {links,[<0.1244.0>]}, {dictionary, [{last_stats_report,{1396,247466,254580}}]}, {trap_exit,false}, {status,waiting}, {heap_size,17731}, {stack_size,41}, {reductions,1927565478}]}]]}} [Mon, 31 Mar 2014 06:31:12 GMT] [error] [<0.98.0>] {error_report,<0.35.0>, {<0.98.0>,supervisor_report, [{supervisor,{local,couch_replicator_job_sup}}, {errorContext,child_terminated}, {reason, {worker_died,<0.1256.0>, {{badmatch, <<27,109,21,85,240,3,44,180,90,71,64,92,107,215,185, 43>>}, [{couch_stream,foldl,6, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_stream.erl"}, {line,96}]}, {couch_doc,att_to_bin,1, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,385}]}, {couch_doc,'-to_json_attachments/4-fun-0-',4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,123}]}, {lists,map,2,[{file,"lists.erl"},{line,1224}]}, {couch_doc,to_json_attachments,4, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,101}]}, {couch_doc,doc_to_json_obj,2, [{file, "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_doc.erl"}, {line,153}]}, {couch_replicator_worker,maybe_flush_docs,3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,411}]}, {couch_replicator_worker, '-local_process_batch/6-fun-1-',3, [{file, "c:/cygwin/relax/APACHE~1.0/src/COUCH_~1/src/couch_replicator_worker.erl"}, {line,263}]}]}}}, {offender, [{pid,<0.1244.0>}, {name,"043a8e83493ef0bc93cdd6a8129450c4+continuous"}, {mfargs,{gen_server,start_link,undefined}}, {restart_type,temporary}, {shutdown,250}, {child_type,worker}]}]}} [Mon, 31 Mar 2014 08:49:25 GMT] [info] [<0.122.0>] 107.252.61.203 - - GET /_utils/index.html 200 -- This message was sent by Atlassian JIRA (v6.2#6252)