activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Broker leaks FDs - Too many open files
Date Mon, 19 Aug 2013 20:18:19 GMT
Yah, give that a try (as seen here
https://issues.apache.org/jira/browse/AMQ-1739).
Could also have a look at
http://activemq.apache.org/maven/apidocs/org/apache/activemq/transport/WriteTimeoutFilter.htmlper
this jira
https://issues.apache.org/jira/browse/AMQ-1993


On Mon, Aug 19, 2013 at 12:51 PM, Jerry Cwiklik <cwiklik@us.ibm.com> wrote:

> Our production broker (v.5.6.0) keeps dying while in heavy use. The broker
> log is filled with:
>
> 2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector
> - Could not accept connection : java.net.SocketException: Too many open
> files
>
> 2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector
> - Could not accept connection : java.net.SocketException: Too many open
> files
>
> 2013-07-28 00:04:08,264 [teTimeout=45000] ERROR TransportConnector
> - Could not accept connection : java.net.SocketException: Too many open
> files
>
> This is logged at such a rapid rate that the logs roll and hide the initial
> error/warning. We capture open fd of the broker's process and notice that
> when the broker starts to croak the open fd count just explodes. Here is
> part of the open fd log. The first column shows broker's open fds and each
> line is logged every 60 secs.
>
>    1284   12569  160194   -- normal count
>    1294   12669  161438
>    1305   12779  162812
>    1318   12909  164426
>    1328   13009  165658   --------- FD explosion
>    1393   13659  173816
>    1528   15009  190748
>    1611   15839  201152
>    1701   16739  212419
>    1951   19239  243520
>    2310   22830  290374
>    2667   26399  332362
>    3013   29859  375262
>    3369   33422  422638
>    3729   37019  464017
>    4111   40841  515342
>    4484   44570  561933
>    4870   48432  609992
>    5249   52219  652157
>    5634   56071  705356
>    6019   59919  747457
>    6484   64571  811476
>    6892   68652  862375
>    7307   72802  914122
>    7727   77002  966555
>    8129   81022 1016717
>    8336   83090 1042601
>    8336   83090 1042584
>    8336   83090 1042583
>
> It normally shows ~1300 fds and this is more or less constant overtime, but
> eventually it rapidly increases to 8336 and the broker becomes unusable.
> The
> ulimit is set to 4094. The netstat shows a ton of sockets in CLOSE_WAIT
> suggesting that the broker is not closing its side of a socket.
>
> I found related open issue
>
> https://issues.apache.org/jira/browse/AMQ-4531?page=com.atlassian.jirafisheyeplugin:fisheye-issuepanel
>
> This Jira states that the problem surfaces in 5.8.0 and when
> maximumConnections is set. We dont use this setting and we run with an
> older
> version of AMQ. Any ideas how to deal with this? Would closeAsync=false
> have
> any effect?
>
> JC
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-leaks-FDs-Too-many-open-files-tp4670496.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message