activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arjen van der Meijden (JIRA)" <>
Subject [jira] Updated: (AMQ-1739) ActiveMQ 5.1.0 runs out of file descriptors with lots of 'CLOSE_WAIT' sockets
Date Tue, 27 May 2008 11:41:55 GMT


Arjen van der Meijden updated AMQ-1739:

    Attachment: stomp-overload-producer.tgz

The issue - or a similar one - reoccured yesterday.

It appears ActiveMQ 5.1/5.2-snapshot copes much worse with queues that are filled faster than
consumed (or not consumed at all) than 5.0.

With 5.0 I could fill up two queues to over 80k small messages, but 5.2-snapshot refused to
accept connections after about 5120 message on one queue and 3690 on the other.

The attached tgz is a set of two php5 scripts. Just extract it somewhere and adjust the $brokerUri
in 'producerserver.php' to fit your needs. Then you should be able to run it and get an overloaded
activemq within a minute or so.

Running is simple, make sure you have a CLI-version of php 5 (in debian that is 'apt-get install
php5-cli') and run: 'php producerserver.php'

That script will then include the default Stomp-implementation for php and connect to ActiveMQ
using Stomp. It forks off $max child-processes, until you stop the script, which send two
messages to two seperate queues.  If you need any special connection-parameters, have a look
at the Stomp.php StompConnection's 'getInstance'-method, but the above should work with a
default ActiveMQ-install.

Our ActiveMQ is started with a slightly adjusted activemq.xml (ssl and xmmp connections are
not started, camel-section is empty, resource management is commented out) but mostly left
to the defaults. Our ACTIVEMQ_OPTS = "-XX:+UseParallelOldGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Xms1024M -Xmx1024M -Dorg.apache.activemq.UseDedicatedTaskRunner=true"

Although normally you'd have at least one consumer for those queue's, its easier and faster
to reproduce the issue without one. That way you should get 'ERROR TransportConnector    
        - Could not accept connection : Too many open files' pretty fast.

> ActiveMQ 5.1.0 runs out of file descriptors with lots of 'CLOSE_WAIT' sockets
> -----------------------------------------------------------------------------
>                 Key: AMQ-1739
>                 URL:
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.1.0
>         Environment: We have a single broker with no special network-stuff. Our broker-system
has two single core Opterons, 8GB of memory, plenty of I/O and runs a recent 64bit debian
with 2.6.21 kernel.
> Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 10.0-b22, mixed mode)
> We left most of the activemq.xml-configuration as-is and adjusted the start-up script
to run with 2GB heap size and parallel garbage collector, which was more or less needed for
5.0 and left for 5.1 in the start-up script.
>            Reporter: Arjen van der Meijden
>            Assignee: Rob Davies
>            Priority: Blocker
>         Attachments: stomp-overload-producer.tgz
> We have no idea why or when, but within a few days after start-up, ActiveMQ suddenly
runs out of file descriptors (we've raised the limit to 10240). According to lsof it has lots
of sockets which are in CLOSE_WAIT when that happens. As soon as that happened once, it would
re-occur within a few hours. This behavior did not happen with ActiveMQ 5.0.
> We have five queues, all with only one consumer. All consumption and production is via
the Stomp-interface using PHP-clients. Three of those queues get up to 50-100 messages/second
in peak moments, while the consumers adjust their own consumption rate to the systems load
(normally its maxed to about 50-150/sec). So in high-load moments on the consumers, the queues
can grow to a few thousand messages, normally the queues are emptied as soon as a message
occurs. Those five consumers stay connected indefinitely.
> The messages are all quite small (at most 1 KB or so) and come from 5 web servers. For
each web page-request (about 2-3M/day) a connection is made to ActiveMQ via Stomp and at least
one message is sent to ActiveMQ, for most requests two are sent to the two most active queues.
For all request a new connection is made and at most 4 stomp-messages are sent to ActiveMQ
(connect, two messages, disconnect), since apache+php does not allow useful reuse of sockets
and similar resources.So 
> So the connection-rate is about the same as the highest message rate on any of the queues
(so 50-100connects/second).
> When the high amount of sockets in CLOSE_WAIT occurs, we manually disable the producers
and the sockets disappear gradually. After that the amount of sockets stays around 180-190
(mostly opened jars), but seams to re-increase more easily than when ActiveMQ is restarted.
I have not checked if anything special happens on the web servers or databases, since their
producer and consumer implementation has not changed since 5.0.
> What I did notice is that the memory-consumption increases heavily prior to running out
of descriptors, and the consumption re-increases way to fast compared to before 11:45u:

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message