incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Stockton <chrisstockto...@gmail.com>
Subject Trying to get 5000+ databases replicating, no success thus far.
Date Thu, 15 Jul 2010 23:18:54 GMT
Hello,

I have gone through a good amount of research and effort trying to get
a large amount of databases replicating, with a good amount of help
from this list. I have hit a road block and am unable to get more then
2000+ databases replicating.

Something tells me [1] might be part of my issue. I have downloaded
and installed couchdb-0.11.1, I made sure that patch was present in
the source. I included the limits in the INI file but they do not seem
to be respected, or I am hitting a different kind of limit. Here [2]
is the errors that couch starts throwing when I get to roughly 2048~
(the default number of allowed connections).

These exist in my couchdb init script:
ulimit -n 1000000
export ERL_MAX_PORTS=100000
export ERL_MAX_ETS_TABLES=10000

This is the init script start command:
daemon --user couchdb "$exec -b -p
/<dir>/couchdb-0.11.1-test/couch.pid -a
/<dir>/couchdb-0.11.1-test/local.ini +P 65536"

I am certain the following local.ini is being parsed, as I run on a
non-standard port, in addition I am certain the path is correct:
[couchdb]
os_process_timeout=60000
max_dbs_open = 100000

[httpd]
max_connections = 9000
port = 59840
bind_address = 0.0.0.0

At this point I am looking for some expertise from other people who
have dealt with large-scale replication.

Kind Regards,

-Chris

[1] https://issues.apache.org/jira/browse/COUCHDB-705
[2] ---

[Thu, 15 Jul 2010 23:03:53 GMT] [error] [emulator] ** Too many db tables **



[Thu, 15 Jul 2010 23:03:53 GMT] [error] [<0.21959.3>] {error_report,<0.28.0>,
    {<0.21959.3>,crash_report,
     [[{pid,<0.21959.3>},
       {registered_name,[]},
       {error_info,
           {exit,
               {system_limit,
                   [{ets,new,[replication_stats,[set,private]]},
                    {couch_rep,do_init,1},
                    {couch_rep,init,1},
                    {gen_server,init_it,6},
                    {proc_lib,init_p_do_apply,3}]},
               [{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},
       {initial_call,{couch_rep,init,['Argument__1']}},
       {ancestors,
           [couch_rep_sup,couch_primary_services,couch_server_sup,<0.29.0>]},
       {messages,[]},
       {links,[<0.22081.3>,<0.22182.3>,<0.22268.3>,<0.22166.3>,<0.78.0>]},
       {dictionary,[]},
       {trap_exit,true},
       {status,running},
       {heap_size,2584},
       {stack_size,23},
       {reductions,2097}],
      [{neighbour,
           [{pid,<0.22268.3>},
            {registered_name,[]},
            {initial_call,{erlang,apply,2}},
            {current_function,{erlang,apply,2}},
            {ancestors,[]},
            {messages,[]},
            {links,[<0.21959.3>]},
            {dictionary,[]},
            {trap_exit,false},
            {status,runnable},
            {heap_size,377},
            {stack_size,0},
            {reductions,0}]}]]}}

[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] ** Generic
server <0.22166.3> terminating
** Last message in was {'EXIT',<0.21959.3>,
                           {system_limit,
                               [{ets,new,[replication_stats,[set,private]]},
                                {couch_rep,do_init,1},
                                {couch_rep,init,1},
                                {gen_server,init_it,6},
                                {proc_lib,init_p_do_apply,3}]}}
** When Server state == {state,<0.22167.3>,nil,
                            {db,<0.21978.3>,<0.22062.3>,nil,
                                <<"1279235033964957">>,<0.21967.3>,
                                <0.22063.3>,
                                {db_header,5,0,0,nil,nil,nil,0,nil,nil,1000},
                                0,
                                {btree,<0.21967.3>,nil,
                                    #Fun<couch_db_updater.7.82129660>,
                                    #Fun<couch_db_updater.8.42953822>,
                                    #Fun<couch_btree.5.124754102>,
                                    #Fun<couch_db_updater.9.115326703>},
                                {btree,<0.21967.3>,nil,
                                    #Fun<couch_db_updater.10.103072508>,
                                    #Fun<couch_db_updater.11.104248294>,
                                    #Fun<couch_btree.5.124754102>,
                                    #Fun<couch_db_updater.12.125559248>},
                                {btree,<0.21967.3>,nil,
                                    #Fun<couch_btree.0.83553141>,
                                    #Fun<couch_btree.1.30790806>,
                                    #Fun<couch_btree.2.124754102>,nil},
                                0,<<"db_2213">>,

"/var/lib/mysql_data/couchdb-0.11.1-test/var/lib/couchdb/db_2213.couch",
                                [],[],nil,
                                {user_ctx,<<"network">>,
                                    [<<"_admin">>],
                                    <<"{couch_httpd_auth,
default_authentication_handler}">>},
                                nil,1000,
                                [before_header,after_header,on_file_open]},
                            <0.21959.3>,false,0,
                            {<0.22184.3>,#Ref<0.0.2.116318>},
                            {[],[]},
                            0,0,0}
** Reason for termination ==
** {system_limit,[{ets,new,[replication_stats,[set,private]]},
                  {couch_rep,do_init,1},
                  {couch_rep,init,1},
                  {gen_server,init_it,6},
                  {proc_lib,init_p_do_apply,3}]}


[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] {error_report,<0.28.0>,
    {<0.22166.3>,crash_report,
     [[{pid,<0.22166.3>},
       {registered_name,[]},
       {error_info,
           {exit,
               {system_limit,
                   [{ets,new,[replication_stats,[set,private]]},
                    {couch_rep,do_init,1},
                    {couch_rep,init,1},
                    {gen_server,init_it,6},
                    {proc_lib,init_p_do_apply,3}]},
               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
       {initial_call,{couch_rep_missing_revs,init,['Argument__1']}},
       {ancestors,
           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
            <0.29.0>]},
       {messages,[]},
       {links,[<0.22167.3>]},
       {dictionary,[]},
       {trap_exit,true},
       {status,running},
       {heap_size,233},
       {stack_size,23},
       {reductions,144}],
      []]}}

[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] ** Generic
server <0.22182.3> terminating
** Last message in was {'EXIT',<0.21959.3>,
                           {system_limit,
                               [{ets,new,[replication_stats,[set,private]]},
                                {couch_rep,do_init,1},
                                {couch_rep,init,1},
                                {gen_server,init_it,6},
                                {proc_lib,init_p_do_apply,3}]}}
** When Server state == {state,<0.21959.3>,
                            {http_db,

"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462/",
                                [],[],
                                [{"User-Agent","CouchDB/0.11.1"},
                                 {"Accept","application/json"},
                                 {"Accept-Encoding","gzip"}],
                                [],get,nil,
                                [{response_format,binary},
                                 {inactivity_timeout,30000}],
                                10,500,nil},
                            <0.22166.3>,<0.22184.3>,[],0,
                            {[],[]},
                            nil,false,0,nil,[],[]}
** Reason for termination ==
** {system_limit,[{ets,new,[replication_stats,[set,private]]},
                  {couch_rep,do_init,1},
                  {couch_rep,init,1},
                  {gen_server,init_it,6},
                  {proc_lib,init_p_do_apply,3}]}


[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] {error_report,<0.28.0>,
    {<0.22182.3>,crash_report,
     [[{pid,<0.22182.3>},
       {registered_name,[]},
       {error_info,
           {exit,
               {system_limit,
                   [{ets,new,[replication_stats,[set,private]]},
                    {couch_rep,do_init,1},
                    {couch_rep,init,1},
                    {gen_server,init_it,6},
                    {proc_lib,init_p_do_apply,3}]},
               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
       {initial_call,{couch_rep_reader,init,['Argument__1']}},
       {ancestors,
           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
            <0.29.0>]},
       {messages,[]},
       {links,[<0.22184.3>]},
       {dictionary,[]},
       {trap_exit,true},
       {status,running},
       {heap_size,987},
       {stack_size,23},
       {reductions,282}],
      [{neighbour,
           [{pid,<0.22184.3>},
            {registered_name,[]},
            {initial_call,{erlang,apply,2}},
            {current_function,{gen,wait_resp_mon,3}},
            {ancestors,[]},
            {messages,
                [{'DOWN',#Ref<0.0.2.116318>,process,<0.22166.3>,
                     {system_limit,
                         [{ets,new,[replication_stats,[set,private]]},
                          {couch_rep,do_init,1},
                          {couch_rep,init,1},
                          {gen_server,init_it,6},
                          {proc_lib,init_p_do_apply,3}]}}]},
            {links,[<0.22182.3>]},
            {dictionary,[]},
            {trap_exit,false},
            {status,runnable},
            {heap_size,377},
            {stack_size,15},
            {reductions,14}]}]]}}

[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.31207.0>] Uncaught error
in HTTP request: {error,
                                 {case_clause,
                                  {error,
                                   {{system_limit,
                                     [{ets,new,
                                       [replication_stats,[set,private]]},
                                      {couch_rep,do_init,1},
                                      {couch_rep,init,1},
                                      {gen_server,init_it,6},
                                      {proc_lib,init_p_do_apply,3}]},
                                    {child,undefined,

"a0bd902c5803e5fb8e59918c8805d776+continuous+create_target",
                                     {gen_server,start_link,
                                      [couch_rep,
                                       ["a0bd902c5803e5fb8e59918c8805d776",
                                        {[{<<"source">>,

<<"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462">>},
                                          {<<"target">>,<<"db_2213">>},
                                          {<<"continuous">>,true},
                                          {<<"create_target">>,true},
                                          {<<"cancel">>,false}]},
                                        {user_ctx,<<"network">>,
                                         [<<"_admin">>],
                                         <<"{couch_httpd_auth,
default_authentication_handler}">>}],
                                       []]},
                                     temporary,1,worker,
                                     [couch_rep]}}}}}

[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] ** Generic
server <0.22081.3> terminating
** Last message in was {'EXIT',<0.21959.3>,
                           {system_limit,
                               [{ets,new,[replication_stats,[set,private]]},
                                {couch_rep,do_init,1},
                                {couch_rep,init,1},
                                {gen_server,init_it,6},
                                {proc_lib,init_p_do_apply,3}]}}
** When Server state == {state,nil,nil,
                            [<0.21959.3>,
                             {http_db,

"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462/",
                                 [],[],
                                 [{"User-Agent","CouchDB/0.11.1"},
                                  {"Accept","application/json"},
                                  {"Accept-Encoding","gzip"}],
                                 [],get,nil,
                                 [{response_format,binary},
                                  {inactivity_timeout,30000}],
                                 10,500,nil},
                             0,
                             [{<<"source">>,

<<"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462">>},
                              {<<"target">>,<<"db_2213">>},
                              {<<"continuous">>,true},
                              {<<"create_target">>,true},
                              {<<"cancel">>,false}]],
                            0,<0.22165.3>,
                            {1279,235033,989017},
                            false,0,<<>>,
                            {<0.22167.3>,#Ref<0.0.2.116315>},
                            {[],[]}}
** Reason for termination ==
** {system_limit,[{ets,new,[replication_stats,[set,private]]},
                  {couch_rep,do_init,1},
                  {couch_rep,init,1},
                  {gen_server,init_it,6},
                  {proc_lib,init_p_do_apply,3}]}


[Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] {error_report,<0.28.0>,
    {<0.22081.3>,crash_report,
     [[{pid,<0.22081.3>},
       {registered_name,[]},
       {error_info,
           {exit,
               {system_limit,
                   [{ets,new,[replication_stats,[set,private]]},
                    {couch_rep,do_init,1},
                    {couch_rep,init,1},
                    {gen_server,init_it,6},
                    {proc_lib,init_p_do_apply,3}]},
               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
       {initial_call,{couch_rep_changes_feed,init,['Argument__1']}},
       {ancestors,
           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
            <0.29.0>]},
       {messages,
           [{ibrowse_async_response,
                {1279,235033,989017},
                {error,closing_on_request}},
            {'EXIT',<0.22165.3>,normal}]},
       {links,[]},
       {dictionary,[]},
       {trap_exit,true},
       {status,running},
       {heap_size,987},
       {stack_size,23},
       {reductions,903}],
      []]}}

[Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] Stacktrace:
[{couch_rep,start_replication_server,1},
             {couch_rep,replicate,2},
             {couch_httpd_misc_handlers,handle_replicate_req,1},
             {couch_httpd,handle_request_int,5},
             {mochiweb_http,headers,5},
             {proc_lib,init_p_do_apply,3}]

[Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] 172.19.68.122 - -
'POST' /_replicate 500

[Thu, 15 Jul 2010 23:03:54 GMT] [debug] [<0.31207.0>] httpd 500 error response:
 {"error":"case_clause","reason":"{error,\n    {{system_limit,\n
  [{ets,new,[replication_stats,[set,private]]},\n
{couch_rep,do_init,1},\n          {couch_rep,init,1},\n
{gen_server,init_it,6},\n          {proc_lib,init_p_do_apply,3}]},\n
  {child,undefined,\n
\"a0bd902c5803e5fb8e59918c8805d776+continuous+create_target\",\n
  {gen_server,start_link,\n             [couch_rep,\n
[\"a0bd902c5803e5fb8e59918c8805d776\",\n
{[{<<\"source\">>,\n
<<\"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462\">>},\n
                {<<\"target\">>,<<\"db_2213\">>},\n
{<<\"continuous\">>,true},\n
{<<\"create_target\">>,true},\n
{<<\"cancel\">>,false}]},\n               {user_ctx,<<\"network\">>,\n
                  [<<\"_admin\">>],\n
<<\"{couch_httpd_auth, default_authentication_handler}\">>}],\n
      []]},\n         temporary,1,worker,\n         [couch_rep]}}}"}

---

Mime
View raw message