activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerry Cwiklik <cwik...@us.ibm.com>
Subject Broker leaks FDs - Too many open files
Date Mon, 19 Aug 2013 19:51:00 GMT
Our production broker (v.5.6.0) keeps dying while in heavy use. The broker
log is filled with:

2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector            
- Could not accept connection : java.net.SocketException: Too many open
files

2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector            
- Could not accept connection : java.net.SocketException: Too many open
files

2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector            
- Could not accept connection : java.net.SocketException: Too many open
files
 
This is logged at such a rapid rate that the logs roll and hide the initial
error/warning. We capture open fd of the broker's process and notice that
when the broker starts to croak the open fd count just explodes. Here is
part of the open fd log. The first column shows broker's open fds and each
line is logged every 60 secs.

   1284   12569  160194   -- normal count
   1294   12669  161438
   1305   12779  162812
   1318   12909  164426
   1328   13009  165658   --------- FD explosion
   1393   13659  173816
   1528   15009  190748
   1611   15839  201152
   1701   16739  212419
   1951   19239  243520
   2310   22830  290374
   2667   26399  332362
   3013   29859  375262
   3369   33422  422638
   3729   37019  464017
   4111   40841  515342
   4484   44570  561933
   4870   48432  609992
   5249   52219  652157
   5634   56071  705356
   6019   59919  747457
   6484   64571  811476
   6892   68652  862375
   7307   72802  914122
   7727   77002  966555
   8129   81022 1016717
   8336   83090 1042601
   8336   83090 1042584
   8336   83090 1042583
 
It normally shows ~1300 fds and this is more or less constant overtime, but
eventually it rapidly increases to 8336 and the broker becomes unusable. The
ulimit is set to 4094. The netstat shows a ton of sockets in CLOSE_WAIT
suggesting that the broker is not closing its side of a socket.

I found related open issue
https://issues.apache.org/jira/browse/AMQ-4531?page=com.atlassian.jirafisheyeplugin:fisheye-issuepanel

This Jira states that the problem surfaces in 5.8.0 and when
maximumConnections is set. We dont use this setting and we run with an older
version of AMQ. Any ideas how to deal with this? Would closeAsync=false have
any effect? 

JC




--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-leaks-FDs-Too-many-open-files-tp4670496.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message