activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mickhayes <mickha...@gmail.com>
Subject AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks
Date Fri, 08 Jul 2011 11:43:17 GMT
Hi folks,

This (after plenty searching) is not a request for help so much as an early
start/nucleation point for this issue: somewhere to find and hang more info.
[Checked: We're not using Stomp + NIO; We do not have the symptoms of
AMQ-2999 (don't use peer transport.); There is no thread leak]

No reproducible test case yet, but when isolated, we will update. First seen
on migration to 5.4.1 from 4.1.1, but various changes in amq xml
configuration were made at the same time. We have some 4.1.1 clients
remaining, and understand that the combination of 4.1.1 client & 5.4.1
broker is "tested by the community" i.e. not tested by the Apache project.

Environment: Solaris 10 u9,  Java HotSpot(TM) 64-Bit Server VM version
17.0-b16.
Single, standalone broker 5.4.1 running as SMF service. Multiple clients on
same machine and others.
We have leaked up to 2000 ticots FDs so far on the busiest of our test
deployments - and it's only running approx. 5K messages per minute at the
moment. [During load test we would put of the order of 1M messages per
minute through the broker.]

Symptom: new FDs of the form (as seen from output of "lsof -p AMQpid | grep
ticots" ):
COMMAND  PID   USER   FD   TYPE        DEVICE    SIZE/OFF     NODE NAME
*java    *12345 some_user   someFDnum  *unix       **105*,xyz         *0t0*
*55050244* */devices/pseudo/tl@0:ticots->(socketpair:* 0xsomenum)
(0xsomenum)

I've bolded the non-varying parts.

This leak is suspected to be triggered by particular JMS/Openwire traffic.
We now have full DEBUG logging on and are trying to corellate the creation
of a new ticots FD with particular events in the log - and perhaps linking
the leak to a particular client.
Also using Dtrace to look at FD creation and disposal.

Changes from previous broker deployment, where the issue was not seen:
ActiveMQ version: 4.1.1 -> 5.4.1
TCP transport -> NIO transport (broker only change)
discardingDLQ plugin introduced.

All the best,
Mick Hayes.





-----
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA 

--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3653804.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message