incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dionne <dio...@dionne-associates.com>
Subject Re: Trying to get 5000+ databases replicating, no success thus far.
Date Thu, 15 Jul 2010 23:27:35 GMT
I've seen this message before when using mnesia, the default for ets tables is 1400. You might
try setting the environment variable:

ERL_MAX_ETS_TABLES

to something higher



On Jul 15, 2010, at 7:18 PM, Chris Stockton wrote:

> Hello,
> 
> I have gone through a good amount of research and effort trying to get
> a large amount of databases replicating, with a good amount of help
> from this list. I have hit a road block and am unable to get more then
> 2000+ databases replicating.
> 
> Something tells me [1] might be part of my issue. I have downloaded
> and installed couchdb-0.11.1, I made sure that patch was present in
> the source. I included the limits in the INI file but they do not seem
> to be respected, or I am hitting a different kind of limit. Here [2]
> is the errors that couch starts throwing when I get to roughly 2048~
> (the default number of allowed connections).
> 
> These exist in my couchdb init script:
> ulimit -n 1000000
> export ERL_MAX_PORTS=100000
> export ERL_MAX_ETS_TABLES=10000
> 
> This is the init script start command:
> daemon --user couchdb "$exec -b -p
> /<dir>/couchdb-0.11.1-test/couch.pid -a
> /<dir>/couchdb-0.11.1-test/local.ini +P 65536"
> 
> I am certain the following local.ini is being parsed, as I run on a
> non-standard port, in addition I am certain the path is correct:
> [couchdb]
> os_process_timeout=60000
> max_dbs_open = 100000
> 
> [httpd]
> max_connections = 9000
> port = 59840
> bind_address = 0.0.0.0
> 
> At this point I am looking for some expertise from other people who
> have dealt with large-scale replication.
> 
> Kind Regards,
> 
> -Chris
> 
> [1] https://issues.apache.org/jira/browse/COUCHDB-705
> [2] ---
> 
> [Thu, 15 Jul 2010 23:03:53 GMT] [error] [emulator] ** Too many db tables **
> 
> 
> 
> [Thu, 15 Jul 2010 23:03:53 GMT] [error] [<0.21959.3>] {error_report,<0.28.0>,
>    {<0.21959.3>,crash_report,
>     [[{pid,<0.21959.3>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {system_limit,
>                   [{ets,new,[replication_stats,[set,private]]},
>                    {couch_rep,do_init,1},
>                    {couch_rep,init,1},
>                    {gen_server,init_it,6},
>                    {proc_lib,init_p_do_apply,3}]},
>               [{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},
>       {initial_call,{couch_rep,init,['Argument__1']}},
>       {ancestors,
>           [couch_rep_sup,couch_primary_services,couch_server_sup,<0.29.0>]},
>       {messages,[]},
>       {links,[<0.22081.3>,<0.22182.3>,<0.22268.3>,<0.22166.3>,<0.78.0>]},
>       {dictionary,[]},
>       {trap_exit,true},
>       {status,running},
>       {heap_size,2584},
>       {stack_size,23},
>       {reductions,2097}],
>      [{neighbour,
>           [{pid,<0.22268.3>},
>            {registered_name,[]},
>            {initial_call,{erlang,apply,2}},
>            {current_function,{erlang,apply,2}},
>            {ancestors,[]},
>            {messages,[]},
>            {links,[<0.21959.3>]},
>            {dictionary,[]},
>            {trap_exit,false},
>            {status,runnable},
>            {heap_size,377},
>            {stack_size,0},
>            {reductions,0}]}]]}}
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] ** Generic
> server <0.22166.3> terminating
> ** Last message in was {'EXIT',<0.21959.3>,
>                           {system_limit,
>                               [{ets,new,[replication_stats,[set,private]]},
>                                {couch_rep,do_init,1},
>                                {couch_rep,init,1},
>                                {gen_server,init_it,6},
>                                {proc_lib,init_p_do_apply,3}]}}
> ** When Server state == {state,<0.22167.3>,nil,
>                            {db,<0.21978.3>,<0.22062.3>,nil,
>                                <<"1279235033964957">>,<0.21967.3>,
>                                <0.22063.3>,
>                                {db_header,5,0,0,nil,nil,nil,0,nil,nil,1000},
>                                0,
>                                {btree,<0.21967.3>,nil,
>                                    #Fun<couch_db_updater.7.82129660>,
>                                    #Fun<couch_db_updater.8.42953822>,
>                                    #Fun<couch_btree.5.124754102>,
>                                    #Fun<couch_db_updater.9.115326703>},
>                                {btree,<0.21967.3>,nil,
>                                    #Fun<couch_db_updater.10.103072508>,
>                                    #Fun<couch_db_updater.11.104248294>,
>                                    #Fun<couch_btree.5.124754102>,
>                                    #Fun<couch_db_updater.12.125559248>},
>                                {btree,<0.21967.3>,nil,
>                                    #Fun<couch_btree.0.83553141>,
>                                    #Fun<couch_btree.1.30790806>,
>                                    #Fun<couch_btree.2.124754102>,nil},
>                                0,<<"db_2213">>,
> 
> "/var/lib/mysql_data/couchdb-0.11.1-test/var/lib/couchdb/db_2213.couch",
>                                [],[],nil,
>                                {user_ctx,<<"network">>,
>                                    [<<"_admin">>],
>                                    <<"{couch_httpd_auth,
> default_authentication_handler}">>},
>                                nil,1000,
>                                [before_header,after_header,on_file_open]},
>                            <0.21959.3>,false,0,
>                            {<0.22184.3>,#Ref<0.0.2.116318>},
>                            {[],[]},
>                            0,0,0}
> ** Reason for termination ==
> ** {system_limit,[{ets,new,[replication_stats,[set,private]]},
>                  {couch_rep,do_init,1},
>                  {couch_rep,init,1},
>                  {gen_server,init_it,6},
>                  {proc_lib,init_p_do_apply,3}]}
> 
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] {error_report,<0.28.0>,
>    {<0.22166.3>,crash_report,
>     [[{pid,<0.22166.3>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {system_limit,
>                   [{ets,new,[replication_stats,[set,private]]},
>                    {couch_rep,do_init,1},
>                    {couch_rep,init,1},
>                    {gen_server,init_it,6},
>                    {proc_lib,init_p_do_apply,3}]},
>               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
>       {initial_call,{couch_rep_missing_revs,init,['Argument__1']}},
>       {ancestors,
>           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
>            <0.29.0>]},
>       {messages,[]},
>       {links,[<0.22167.3>]},
>       {dictionary,[]},
>       {trap_exit,true},
>       {status,running},
>       {heap_size,233},
>       {stack_size,23},
>       {reductions,144}],
>      []]}}
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] ** Generic
> server <0.22182.3> terminating
> ** Last message in was {'EXIT',<0.21959.3>,
>                           {system_limit,
>                               [{ets,new,[replication_stats,[set,private]]},
>                                {couch_rep,do_init,1},
>                                {couch_rep,init,1},
>                                {gen_server,init_it,6},
>                                {proc_lib,init_p_do_apply,3}]}}
> ** When Server state == {state,<0.21959.3>,
>                            {http_db,
> 
> "http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462/",
>                                [],[],
>                                [{"User-Agent","CouchDB/0.11.1"},
>                                 {"Accept","application/json"},
>                                 {"Accept-Encoding","gzip"}],
>                                [],get,nil,
>                                [{response_format,binary},
>                                 {inactivity_timeout,30000}],
>                                10,500,nil},
>                            <0.22166.3>,<0.22184.3>,[],0,
>                            {[],[]},
>                            nil,false,0,nil,[],[]}
> ** Reason for termination ==
> ** {system_limit,[{ets,new,[replication_stats,[set,private]]},
>                  {couch_rep,do_init,1},
>                  {couch_rep,init,1},
>                  {gen_server,init_it,6},
>                  {proc_lib,init_p_do_apply,3}]}
> 
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] {error_report,<0.28.0>,
>    {<0.22182.3>,crash_report,
>     [[{pid,<0.22182.3>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {system_limit,
>                   [{ets,new,[replication_stats,[set,private]]},
>                    {couch_rep,do_init,1},
>                    {couch_rep,init,1},
>                    {gen_server,init_it,6},
>                    {proc_lib,init_p_do_apply,3}]},
>               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
>       {initial_call,{couch_rep_reader,init,['Argument__1']}},
>       {ancestors,
>           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
>            <0.29.0>]},
>       {messages,[]},
>       {links,[<0.22184.3>]},
>       {dictionary,[]},
>       {trap_exit,true},
>       {status,running},
>       {heap_size,987},
>       {stack_size,23},
>       {reductions,282}],
>      [{neighbour,
>           [{pid,<0.22184.3>},
>            {registered_name,[]},
>            {initial_call,{erlang,apply,2}},
>            {current_function,{gen,wait_resp_mon,3}},
>            {ancestors,[]},
>            {messages,
>                [{'DOWN',#Ref<0.0.2.116318>,process,<0.22166.3>,
>                     {system_limit,
>                         [{ets,new,[replication_stats,[set,private]]},
>                          {couch_rep,do_init,1},
>                          {couch_rep,init,1},
>                          {gen_server,init_it,6},
>                          {proc_lib,init_p_do_apply,3}]}}]},
>            {links,[<0.22182.3>]},
>            {dictionary,[]},
>            {trap_exit,false},
>            {status,runnable},
>            {heap_size,377},
>            {stack_size,15},
>            {reductions,14}]}]]}}
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.31207.0>] Uncaught error
> in HTTP request: {error,
>                                 {case_clause,
>                                  {error,
>                                   {{system_limit,
>                                     [{ets,new,
>                                       [replication_stats,[set,private]]},
>                                      {couch_rep,do_init,1},
>                                      {couch_rep,init,1},
>                                      {gen_server,init_it,6},
>                                      {proc_lib,init_p_do_apply,3}]},
>                                    {child,undefined,
> 
> "a0bd902c5803e5fb8e59918c8805d776+continuous+create_target",
>                                     {gen_server,start_link,
>                                      [couch_rep,
>                                       ["a0bd902c5803e5fb8e59918c8805d776",
>                                        {[{<<"source">>,
> 
> <<"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462">>},
>                                          {<<"target">>,<<"db_2213">>},
>                                          {<<"continuous">>,true},
>                                          {<<"create_target">>,true},
>                                          {<<"cancel">>,false}]},
>                                        {user_ctx,<<"network">>,
>                                         [<<"_admin">>],
>                                         <<"{couch_httpd_auth,
> default_authentication_handler}">>}],
>                                       []]},
>                                     temporary,1,worker,
>                                     [couch_rep]}}}}}
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] ** Generic
> server <0.22081.3> terminating
> ** Last message in was {'EXIT',<0.21959.3>,
>                           {system_limit,
>                               [{ets,new,[replication_stats,[set,private]]},
>                                {couch_rep,do_init,1},
>                                {couch_rep,init,1},
>                                {gen_server,init_it,6},
>                                {proc_lib,init_p_do_apply,3}]}}
> ** When Server state == {state,nil,nil,
>                            [<0.21959.3>,
>                             {http_db,
> 
> "http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462/",
>                                 [],[],
>                                 [{"User-Agent","CouchDB/0.11.1"},
>                                  {"Accept","application/json"},
>                                  {"Accept-Encoding","gzip"}],
>                                 [],get,nil,
>                                 [{response_format,binary},
>                                  {inactivity_timeout,30000}],
>                                 10,500,nil},
>                             0,
>                             [{<<"source">>,
> 
> <<"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462">>},
>                              {<<"target">>,<<"db_2213">>},
>                              {<<"continuous">>,true},
>                              {<<"create_target">>,true},
>                              {<<"cancel">>,false}]],
>                            0,<0.22165.3>,
>                            {1279,235033,989017},
>                            false,0,<<>>,
>                            {<0.22167.3>,#Ref<0.0.2.116315>},
>                            {[],[]}}
> ** Reason for termination ==
> ** {system_limit,[{ets,new,[replication_stats,[set,private]]},
>                  {couch_rep,do_init,1},
>                  {couch_rep,init,1},
>                  {gen_server,init_it,6},
>                  {proc_lib,init_p_do_apply,3}]}
> 
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] {error_report,<0.28.0>,
>    {<0.22081.3>,crash_report,
>     [[{pid,<0.22081.3>},
>       {registered_name,[]},
>       {error_info,
>           {exit,
>               {system_limit,
>                   [{ets,new,[replication_stats,[set,private]]},
>                    {couch_rep,do_init,1},
>                    {couch_rep,init,1},
>                    {gen_server,init_it,6},
>                    {proc_lib,init_p_do_apply,3}]},
>               [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
>       {initial_call,{couch_rep_changes_feed,init,['Argument__1']}},
>       {ancestors,
>           [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup,
>            <0.29.0>]},
>       {messages,
>           [{ibrowse_async_response,
>                {1279,235033,989017},
>                {error,closing_on_request}},
>            {'EXIT',<0.22165.3>,normal}]},
>       {links,[]},
>       {dictionary,[]},
>       {trap_exit,true},
>       {status,running},
>       {heap_size,987},
>       {stack_size,23},
>       {reductions,903}],
>      []]}}
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] Stacktrace:
> [{couch_rep,start_replication_server,1},
>             {couch_rep,replicate,2},
>             {couch_httpd_misc_handlers,handle_replicate_req,1},
>             {couch_httpd,handle_request_int,5},
>             {mochiweb_http,headers,5},
>             {proc_lib,init_p_do_apply,3}]
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] 172.19.68.122 - -
> 'POST' /_replicate 500
> 
> [Thu, 15 Jul 2010 23:03:54 GMT] [debug] [<0.31207.0>] httpd 500 error response:
> {"error":"case_clause","reason":"{error,\n    {{system_limit,\n
>  [{ets,new,[replication_stats,[set,private]]},\n
> {couch_rep,do_init,1},\n          {couch_rep,init,1},\n
> {gen_server,init_it,6},\n          {proc_lib,init_p_do_apply,3}]},\n
>  {child,undefined,\n
> \"a0bd902c5803e5fb8e59918c8805d776+continuous+create_target\",\n
>  {gen_server,start_link,\n             [couch_rep,\n
> [\"a0bd902c5803e5fb8e59918c8805d776\",\n
> {[{<<\"source\">>,\n
> <<\"http://network:savetheSQL@g1dlsdbdb002.dev.glbt1.gdg:59845/db_462\">>},\n
>                {<<\"target\">>,<<\"db_2213\">>},\n
> {<<\"continuous\">>,true},\n
> {<<\"create_target\">>,true},\n
> {<<\"cancel\">>,false}]},\n               {user_ctx,<<\"network\">>,\n
>                  [<<\"_admin\">>],\n
> <<\"{couch_httpd_auth, default_authentication_handler}\">>}],\n
>      []]},\n         temporary,1,worker,\n         [couch_rep]}}}"}
> 
> ---


Mime
View raw message