activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Hollerith (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-1873) Stomp connections doesn't seem to be released cleanly (Too many open files error)
Date Fri, 30 Oct 2009 12:44:53 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=55096#action_55096
] 

Alex Hollerith commented on AMQ-1873:
-------------------------------------

config:
<policyEntry queue=">" producerFlowControl="true" memoryLimit="5mb">

setup:
1 perl stomp producer producing into a queue,
  connecting and disconnecting on every post,
  rather low frequency of posts (0.5/minute)
0 consumers

behaviour:
works OK until around 68 messages are in the queue (surely depends on the size of the messages)

after that you get this in the log:
2009-10-29 20:32:05,189 | INFO  | Usage Manager memory limit reached on queue://test.soccerfeeds.queue.
Producers will be throttled to the rate at which messages are removed <...>

And while the activemq service is in that "throttling producers" state you will see CLOSE_WAIT
sockets building up:
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:41519    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:36141    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:45840    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:43793    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:40212    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:44060    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:43776    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:44032    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:43781    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:40200    CLOSE_WAIT
tcp        0      0 ::ffff:10.60.1.51:61613     ::ffff:10.60.1.206:44045    CLOSE_WAIT

You can watch the numbers grow with:
watch --interval=5 'netstat -an |grep tcp |grep 61613 | grep CLOSE_WAIT |wc -l'

Every post increases the number of CLOSE_WAIT sockets by 1. And the sockets will not go away,
the number seems to be steadily growing.

Just consume one single message (we did it via the admin webinterface) and the number of sockets
in CLOSE_WAIT drops to 0 instantly.

[root@bladedist01 activemq]# netstat -an |grep tcp |grep 61613
tcp        0      0 :::61613                    :::*                        LISTEN

Our theory is that activemq does somehow manage to build up sockets in CLOSE_WAIT state while
it is "throttling producers" mode on a given queue until, eventually, the system runs out
of resources (file descriptors in this case).

> Stomp connections doesn't seem to be released cleanly (Too many open files error)
> ---------------------------------------------------------------------------------
>
>                 Key: AMQ-1873
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1873
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.1.0
>         Environment: Linux RHEL 4
> java version "1.5.0_15"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
> Java HotSpot(TM) Server VM (build 1.5.0_15-b04, mixed mode)
>            Reporter: Julien Devemy
>             Fix For: 5.4.0
>
>         Attachments: activemq.xml
>
>
> I'm actually testing ActiveMQ with a python stomp Client (stomp.py http://www.briggs.net.nz/log/projects/stomppy/).
> I'm trying to load the broker with ~200 producers (on ~200 hosts) feeding one TOPIC for
only one consumer.
> Everything is working almost fine until the client connection fails.
> The ActiveMQ log displays :
> ...
> 2008-07-30 09:28:43,665 [localhost:61613] ERROR TransportConnector             - Could
not accept connection : Too many open files
> 2008-07-30 09:28:43,665 [localhost:61613] DEBUG TransportConnector             - Reason:
Too many open files
> java.net.SocketException: Too many open files
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> 	at java.net.ServerSocket.accept(ServerSocket.java:421)
> 	at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:221)
> 	at java.lang.Thread.run(Thread.java:595)
> When I look at the open files (or connections) (with lsof) I see 65446 lines like :
> java    26765 root *066u  sock        0,4          7534034 can't identify protocol
> This seems to indicate that ActiveMQ doesn't totally released the UNIX socket
> I'm attaching the activemq.xml conf file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message