activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From callain_accovia <call...@accovia.com>
Subject Thread/FD Leak in Active MQ?
Date Wed, 10 Aug 2011 20:31:00 GMT
ActiveMQ 5.3.2

java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)

Description:    CentOS release 5.5 (Final)

The problem is this

We see FDs being consumed by the activemq java process:

/usr/sbin/lsof -p 6700 | grep " identify" | wc -l
798

/usr/sbin/lsof -p 6700 | wc -l
1025

Most "leaked" FDs look like this (they don't show up with netstat -a):

...
java    6700 lexo-ext  904u  sock                0,5          1741176200
can't identify protocol
java    6700 lexo-ext  905u  sock                0,5          1741241481
can't identify protocol
...

Now, I also noted the following errors in activemq logs:

 WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_62568
 WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.7_58352
 WARN | Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_9091

and finally, I now have hundreds of threads waiting on something:

~/activemq/apache-activemq-5.3.2$ jstack 6700 | grep "ActiveMQ Connection
Dispatcher" | wc -l
819

which seem to be waiting:

"ActiveMQ Connection Dispatcher: /192.168.235.8:22405" daemon prio=10
tid=0x0ab16800 nid=0x7196 in Object.wait() [0xbcf360
00]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xdd5a0cf8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
        - locked <0xdd5a0cf8> (a java.lang.Object)
        at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

and only 15 "transport" threads:

~/activemq/apache-activemq-5.3.2$ jstack 6700 | grep "ActiveMQ Transport" |
wc -l
15

Now, is there any relationship between the "Failed to register MBean"
message, the number of FDs increasing (up to the per-process limit after a
few days) and the number of ActiveMQ Connection Dispatcher threads that seem
to remain in the system for nothing?

I saw https://issues.apache.org/jira/browse/AMQ-3286 that was similar to
what I can see and there are also a few bugs related to leaking FPs/Threads,
but could not see something that would match all these conditions (ex:
https://issues.apache.org/jira/browse/AMQ-1791).

Thanks for any help.

--
View this message in context: http://activemq.2283324.n4.nabble.com/Thread-FD-Leak-in-Active-MQ-tp3734161p3734161.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message