activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johan Carlquist <jo...@su.se>
Subject ActiveMQ leaks file descriptors
Date Wed, 02 Nov 2016 09:19:48 GMT
We have two brokers configured in a network of brokers.
      <networkConnector
        networkTTL="-1"
        conduitSubscriptions="false"
        decreaseNetworkConsumerPriority="true"
        uri="static:(ssl://${activemq.networkPartner}:61616)"
        userName="${activemq.username}"
        password="${activemq.password}"
        />
    </networkConnectors>

Both stores the messages on local disk using KahaDB.
    <persistenceAdapter>
        <kahaDB directory="${activemq.data}/kahadb"/>
    </persistenceAdapter>

And they both run ActiveMQ-5.13.4 in java 8u45.

Clients are allowed to connect through openwires and amqps:
    <transportConnectors>
      <transportConnector name="nio+ssl" uri="nio+ssl://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
      <transportConnector name="amqp+ssl" uri="amqp+nio+ssl://0.0.0.0:5671?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
    </transportConnectors>

We have about 10 queues and 1 topic configured. The major part of these
messages are sent by Nagios that frequently checks that everything is
fine by sending and receiving messages on all accepted protocols.

Since we setup this configuration, ActiveMQ has leaked file descriptors
and we haven't been able to figure out the cause.

According to `lsof` there is a lot of pipes and anon_inodes open:
[...]
java    33212 root 6462w  FIFO                0,8      0t0 76388774 pipe
java    33212 root 6463r  FIFO                0,8      0t0 76326567 pipe
java    33212 root 6464w  FIFO                0,8      0t0 76326567 pipe
java    33212 root 6465u  0000                0,9        0     6827 anon_inode
java    33212 root 6466r  FIFO                0,8      0t0 76344481 pipe
java    33212 root 6467w  FIFO                0,8      0t0 76344481 pipe
java    33212 root 6468u  0000                0,9        0     6827 anon_inode
java    33212 root 6469u  0000                0,9        0     6827 anon_inode
java    33212 root 6470r  FIFO                0,8      0t0 76372935 pipe
java    33212 root 6471w  FIFO                0,8      0t0 76372935 pipe
java    33212 root 6472u  0000                0,9        0     6827 anon_inode
java    33212 root 6473r  FIFO                0,8      0t0 76514685 pipe
java    33212 root 6474r  FIFO                0,8      0t0 76435445 pipe
java    33212 root 6475r  FIFO                0,8      0t0 76406235 pipe
java    33212 root 6476w  FIFO                0,8      0t0 76406235 pipe
java    33212 root 6477u  0000                0,9        0     6827 anon_inode
java    33212 root 6478w  FIFO                0,8      0t0 76435445 pipe
[...]

Graph of open files growing over time:
https://imgur.com/a/mM7MC

Other ActiveMQ "too many open files" related issues seems to be about
sockets that hangs on "CLOSE_WAIT" but that doesn't seems to be the
problem here.

Any idea how to fix this?

--
jocar

Mime
View raw message