incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcel Bruch <br...@cs.tu-darmstadt.de>
Subject Replication fails with timeout on large database/large documents
Date Fri, 05 Aug 2011 08:01:04 GMT
Hi,

I'm currently trying to get replication work for large databases. Unfortunately, replication
terminates after a small number of documents with some timeouts. Replication for several other
databases with smaller documents and total db size worked w/o any problem. 

Before termination a few documents have been published (46MB). The documents itself are rather
complex and the DB is currently 6.5G in size. Any ideas what's going wrong or how to solve?

Thanks,
Marcel


Log of the source  server shows:

[Fri, 05 Aug 2011 07:46:51 GMT] [info] [<0.8544.0>] 127.0.0.1 - - 'GET' /udc/_design/metaData/_view/modelSpecificationsIncludeAlias?key=%22org.eclipse.swtbot.eclipse.core%22
200
[Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1030.0>] couch_rep_httpc request failed
after 10 retries: http://137.mmm.mmm.mmm:5984/udc/
[Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1009.0>] ** Generic server <0.1009.0>
terminating 
** Last message in was {'EXIT',<0.1030.0>,
                           {http_request_failed,
                               <<"failed to replicate http://137.mmm.mmm.mmm:5984/udc/">>}}
** When Server state == {state,<0.1024.0>,<0.1026.0>,<0.1028.0>,<0.1030.0>,
                            {db,<0.256.0>,<0.257.0>,nil,
                                <<"1312529491031938">>,<0.254.0>,<0.258.0>,
                                {db_header,5,166309,0,
                                    {6931331728,{163315,2988}},
                                    {6931334614,166303},
                                    {6931341991,[]},
                                    0,nil,nil,1000},
                                166309,
                                {btree,<0.254.0>,
                                    {6931331728,{163315,2988}},
                                    #Fun<couch_db_updater.10.19222179>,
                                    #Fun<couch_db_updater.11.21515767>,
                                    #Fun<couch_btree.5.124754102>,
                                    #Fun<couch_db_updater.12.93888648>},
                                {btree,<0.254.0>,
                                    {6931334614,166303},
                                    #Fun<couch_db_updater.13.40165027>,
                                    #Fun<couch_db_updater.14.82810239>,
                                    #Fun<couch_btree.5.124754102>,
                                    #Fun<couch_db_updater.15.104121193>},
                                {btree,<0.254.0>,
                                    {6931341991,[]},
                                    #Fun<couch_btree.0.83553141>,
                                    #Fun<couch_btree.1.30790806>,
                                    #Fun<couch_btree.2.124754102>,nil},
                                166309,<<"udc">>,
                                "/usr/local/var/lib/couchdb/udc.couch",[],[],
                                nil,
                                {user_ctx,null,
                                    [<<"_admin">>],
                                    <<"{couch_httpd_auth, default_authentication_handler}">>},
                                nil,1000,
                                [before_header,after_header,on_file_open],
                                false},
                            {http_db,"http://137.mmm.mmm.mmm:5984/udc/",[],
                                [],
                                [{"User-Agent","CouchDB/1.1.0"},
                                 {"Accept","application/json"},
                                 {"Accept-Encoding","gzip"}],
                                [],get,nil,
                                [{response_format,binary},
                                 {inactivity_timeout,30000}],
                                10,500,nil},
                            false,false,
                            ["f74fbcf318ccd6c8e53c0ffa0b679e74",
                             {[{<<"source">>,<<"udc">>},
                               {<<"target">>,
                                <<"http://137.mmm.mmm.mmm:5984/udc/">>}]},
                             {user_ctx,null,
                                 [<<"_admin">>],
                                 <<"{couch_httpd_auth, default_authentication_handler}">>}],
                            nil,0,[],<<"9d8ddc375c3ca782ead6cbc84d6e611a">>,
                            {doc,
                                <<"_local/f74fbcf318ccd6c8e53c0ffa0b679e74">>,
                                {0,[<<"6">>]},
                                {[{<<"session_id">>,
                                   <<"3d35ed9a1de0e9c66135b54b55971d2c">>},
                                  {<<"source_last_seq">>,732},
                                  {<<"replication_id_version">>,2},
                                  {<<"history">>,
                                   [{[{<<"session_id">>,
                                       <<"3d35ed9a1de0e9c66135b54b55971d2c">>},
                                      {<<"start_time">>,
                                       <<"Fri, 05 Aug 2011 07:28:50 GMT">>},
                                      {<<"end_time">>,
                                       <<"Fri, 05 Aug 2011 07:29:05 GMT">>},
                                      {<<"start_last_seq">>,0},
                                      {<<"end_last_seq">>,732},
                                      {<<"recorded_seq">>,732},
                                      {<<"missing_checked">>,0},
                                      {<<"missing_found">>,4005},
                                      {<<"docs_read">>,2128},
                                      {<<"docs_written">>,733},
                                      {<<"doc_write_failures">>,0}]}]}]},
                                [],false,[]},
                            {doc,
                                <<"_local/f74fbcf318ccd6c8e53c0ffa0b679e74">>,
                                {0,[<<"2">>]},
                                {[]},
                                [],false,[]},
                            "Fri, 05 Aug 2011 07:32:54 GMT",
                            <<"1312529491031938">>,<<"1312530889632808">>,
                            {[{<<"session_id">>,
                               <<"9d8ddc375c3ca782ead6cbc84d6e611a">>},
                              {<<"source_last_seq">>,1056},
                              {<<"replication_id_version">>,2},
                              {<<"history">>,
                               [{[{<<"session_id">>,
                                   <<"9d8ddc375c3ca782ead6cbc84d6e611a">>},
                                  {<<"start_time">>,
                                   <<"Fri, 05 Aug 2011 07:32:54 GMT">>},
                                  {<<"end_time">>,
                                   <<"Fri, 05 Aug 2011 07:33:10 GMT">>},
                                  {<<"start_last_seq">>,0},
                                  {<<"end_last_seq">>,1056},
                                  {<<"recorded_seq">>,1056},
                                  {<<"missing_checked">>,0},
                                  {<<"missing_found">>,4004},
                                  {<<"docs_read">>,2812},
                                  {<<"docs_written">>,1057},
                                  {<<"doc_write_failures">>,0}]}]}]},
                            [{<0.4586.0>,#Ref<0.0.0.36762>},
                             {<0.3631.0>,#Ref<0.0.0.35475>},
                             {<0.137.0>,#Ref<0.0.0.4775>}],
                            false,1056,65587,
                            {[{<<"source">>,<<"udc">>},
                              {<<"target">>,
                               <<"http://137.mmm.mmm.mmm:5984/udc/">>}]},
                            <0.1031.0>,nil}
** Reason for termination == 
** {http_request_failed,<<"failed to replicate http://137.mmm.mmm.mmm:5984/udc/">>}

[Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1009.0>] {error_report,<0.32.0>,
                      {<0.1009.0>,crash_report,
                       [[{initial_call,{couch_rep,init,['Argument__1']}},
                         {pid,<0.1009.0>},
                         {registered_name,[]},
                         {error_info,
                          {exit,
                           {http_request_failed,
                            <<"failed to replicate http://137.mmm.mmm.mmm:5984/udc/">>},
                           [{gen_server,terminate,6},
                            {proc_lib,init_p_do_apply,3}]}},
                         {ancestors,
                          [couch_rep_sup,couch_primary_services,
                           couch_server_sup,<0.33.0>]},
                         {messages,[]},
                         {links,
                          [<0.1028.0>,<0.1031.0>,<0.1024.0>,<0.1026.0>,
                           <0.83.0>]},
                         {dictionary,
                          [{task_status_update,{{1312,529585,68188},0}}]},
                         {trap_exit,true},
                         {status,running},
                         {heap_size,4181},
                         {stack_size,24},
                         {reductions,38989}],
                        [{neighbour,
                          [{pid,<0.1031.0>},
                           {registered_name,[]},
                           {initial_call,
                            {couch_event_sup,init,['Argument__1']}},
                           {current_function,{gen_server,loop,6}},
                           {ancestors,
                            [<0.1009.0>,couch_rep_sup,couch_primary_services,
                             couch_server_sup,<0.33.0>]},
                           {messages,[{'$gen_cast',stop}]},
                           {links,[<0.1009.0>,<0.86.0>]},
                           {dictionary,[]},
                           {trap_exit,false},
                           {status,runnable},
                           {heap_size,233},
                           {stack_size,9},
                           {reductions,36}]}]]}}
[Fri, 05 Aug 2011 07:47:40 GMT] [error] [<0.1026.0>] ** Generic server <0.1026.0>
terminating 
** Last message in was {'EXIT',<0.1009.0>,
                           {http_request_failed,
                               <<"failed to replicate http://137.mmm.mmm.mmm:5984/udc/">>}}
** When Server state == {state,<0.1027.0>,
                               {<0.1027.0>,#Ref<0.0.0.32151>},
                               <0.1009.0>,false,1001,nil,
                               {[{<<"e342b02dabe55a4c06c546ad01e24c90564e3813">>,
                                  5005,
                                  [{2,

[many docs from debug output...]

And finally :

* Reason for termination == 
** {http_request_failed,<<"failed to replicate http://137.mmm.mmm.mmm:5984/udc/">>}



The target Server log entry shows 10 times (or so):

[Fri, 05 Aug 2011 08:09:15 GMT] [debug] [<0.4821.0>] 'POST' /udc/_bulk_docs {1,1} from
"130.mmm.mmm.mmm"
Headers: [{'Accept',"application/json"},
          {'Accept-Encoding',"gzip"},
          {'Content-Length',"43227543"},
          {'Content-Type',"application/json"},
          {'Host',"137.mmm.mmm.mmm:5984"},
          {'User-Agent',"CouchDB/1.1.0"},
          {"X-Couch-Full-Commit","false"}]
[Fri, 05 Aug 2011 08:09:15 GMT] [debug] [<0.4821.0>] OAuth Params: []



-- 
Eclipse Code Recommenders:
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message