incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Herman Chan <herman...@gmail.com>
Subject Re: Help understanding crash log
Date Thu, 01 May 2014 14:43:31 GMT
We do have 1000+ connection to the db, which we are trying to dial down.  However, even with
lower connection, we hit the crash again, this time I was able to get a better log.  You are
right that we are hitting some limit,

before the crash, the log shows that couch is still trying to open up index from a reboot
that we did.  Once it crash, the log start print out with "Index shutdown by monitor".  Is
there any limit parameter that we can increase?

[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Too many processes
[Thu, 01 May 2014 14:28:04 GMT] [error] [emulator] Error in process <0.3672.477> with
exit value: {system_limit,[{erlang,spawn_opt,[proc_lib,init_p,[<0.3672.477>,[],gen,init_it,[
gen_server,<0.3672.477>,<0.3672.477>,couch_db,{<<42 bytes>>,"/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",<0.21556.480>,[{user_ctx,{user_ctx,null,
[<<6 bytes>>],undefined... 


[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] ** Generic server <0.21556.480>
terminating 
** Last message in was {'EXIT',<0.3672.477>,
                        {system_limit,
                         [{erlang,spawn_opt,
                           [proc_lib,init_p,
                            [<0.3672.477>,[],gen,init_it,
                             [gen_server,<0.3672.477>,<0.3672.477>,couch_db,
                              {<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
                               "/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
                               <0.21556.480>,
                               [{user_ctx,
                                 {user_ctx,null,[<<"_admin">>],undefined}}]},
                              []]],
                            [link]]},
                          {proc_lib,start_link,5},
                          {couch_db,start_link,3},
                          {couch_server,'-open_async/5-fun-0-',4}]}}
** When Server state == {file,
                            {file_descriptor,prim_file,
                                {#Port<0.898531>,307709}},
                            1261681}
** Reason for termination == 
** {system_limit,
       [{erlang,spawn_opt,
            [proc_lib,init_p,
             [<0.3672.477>,[],gen,init_it,
              [gen_server,<0.3672.477>,<0.3672.477>,couch_db,
               {<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
                "/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
                <0.21556.480>,
                [{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}]},
               []]],
             [link]]},
        {proc_lib,start_link,5},
        {couch_db,start_link,3},
        {couch_server,'-open_async/5-fun-0-',4}]}

[Thu, 01 May 2014 14:28:04 GMT] [error] [<0.21556.480>] {error_report,<0.31.0>,
                         {<0.21556.480>,crash_report,
                          [[{initial_call,{couch_file,init,['Argument__1']}},
                            {pid,<0.21556.480>},
                            {registered_name,[]},
                            {error_info,
                             {exit,
                              {system_limit,
                               [{erlang,spawn_opt,
                                 [proc_lib,init_p,
                                  [<0.3672.477>,[],gen,init_it,
                                   [gen_server,<0.3672.477>,<0.3672.477>,
                                    couch_db,
                                    {<<"group_370c0635-e593-45ed-ac96-75e6b318cb35">>,
                                     "/usr/local/var/lib/couchdb/group_370c0635-e593-45ed-ac96-75e6b318cb35.couch",
                                     <0.21556.480>,
                                     [{user_ctx,
                                       {user_ctx,null,
                                        [<<"_admin">>],
                                        undefined}}]},
                                    []]],
                                  [link]]},
                                {proc_lib,start_link,5},
                                {couch_db,start_link,3},
                                {couch_server,'-open_async/5-fun-0-',4}]},
                              [{gen_server,terminate,6},
                               {proc_lib,init_p_do_apply,3}]}},
                            {ancestors,[<0.3672.477>]},
                            {messages,[]},
                            {links,[]},
                            {dictionary,[]},
                            {trap_exit,true},
                            {status,running},
                            {heap_size,610},
                            {stack_size,24},
                            {reductions,973}],
                           []]}}
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Index shutdown by monitor notice
for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Index shutdown by monitor notice
for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Index shutdown by monitor notice
for db: group_15ccf331-257d-4b54-b457-997d342816b9 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Index shutdown by monitor notice
for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20971.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa
idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Index shutdown by monitor notice
for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4892.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9
idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Index shutdown by monitor notice
for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Index shutdown by monitor notice
for db: group_ae50933f-de22-4879-9624-b760106060b3 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Index shutdown by monitor notice
for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4883.35>] Closing index for db: group_15ccf331-257d-4b54-b457-997d342816b9
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12040.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00
idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Index shutdown by monitor notice
for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Index shutdown by monitor notice
for db: group_721d99a3-2257-48d0-8a1e-89294874d06e idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.12032.33>] Closing index for db: group_d006a71d-b0de-4d71-b2f7-06abeeb34e00
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Index shutdown by monitor notice
for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27775.43>] Index shutdown by monitor notice
for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4292.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3
idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.4285.4>] Closing index for db: group_ae50933f-de22-4879-9624-b760106060b3
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6010.43>] Index shutdown by monitor notice
for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.6003.43>] Index shutdown by monitor notice
for db: group_7f082ae6-f41d-4a14-a836-2360303b2e9a idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.20956.87>] Closing index for db: group_5747d16f-4b3b-4522-af10-1dc7d0d644aa
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5933.42>] Index shutdown by monitor notice
for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.5940.42>] Index shutdown by monitor notice
for db: group_8c49d7e8-b61e-41e5-a220-11df59b9cce4 idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18842.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17529.33>] Index shutdown by monitor notice
for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/hub
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.18850.44>] Closing index for db: group_721d99a3-2257-48d0-8a1e-89294874d06e
idx: _design/filters sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.17536.33>] Index shutdown by monitor notice
for db: group_98ff493c-63e8-4714-9940-ccea514d4b1d idx: _design/filters
[Thu, 01 May 2014 14:28:04 GMT] [info] [<0.27768.43>] Closing index for db: group_56a0df90-c79e-4863-ae71-2bde3cb0d801
idx: _design/hub sig: "4f6edcabc4b7a6357b714e1391ed93ac"

On 2014-05-01, at 9:18 AM, Adam Kocoloski <kocolosk@apache.org> wrote:

> On May 1, 2014, at 8:47 AM, Interactive Blueprints <p.van.der.eems@interactiveblueprints.nl>
wrote:
> 
>> 2014-05-01 13:14 GMT+02:00 Herman Chan <hermanccw@gmail.com>:
>>> Thanks Adam,
>>> 
>>> It seems like it is happening again, with more info this time.  It looks like
I am hitting some sort of system limit, can anyone point out where to look next?
>> 
>> Just guessing here..
>> What could be is that you hit the max open file limit of your system.
>> With "ulimit -a" you can see the limits on your system.
>> Usually the max open file limit is somewhere around 1024.
>> I noticed that couchdb loves to have a lot of files open simultaneously.
>> 
>> Iin the same shell you start couchdb, right before you start couchdb,
>> you can do a "ulimit -a 4096" (or another large value), this should
>> give coudhb the ability to open more files.
>> 
>> Hope this helps.
>> 
>> Pieter van der Eems
>> Interactive Blueprints
> 
> That's a good thought Pieter, though typically in that case you'll see an 'emfile' error
in the logs. This particular system_limit error (with {erlang, spawn_link, ...} following
it) occurs when the Erlang VM has reached the maximum number of processes it's allowed to
spawn. Judging from the *long* list of processes linked to couch_httpd in this stacktrace
I'd say Herman's client is improperly leaving connections open. Herman, did you intend to
have 1000s of open TCP connections on this server? Regards,
> 
> Adam


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message