activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominik Bieringer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6094) Memory Leak with abnormal disconnecting consumers
Date Fri, 18 Dec 2015 08:30:46 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063688#comment-15063688
] 

Dominik Bieringer commented on AMQ-6094:
----------------------------------------

Also tried it now with 5.12.2 (built with code from GitHub) and this version seems to be affected
by the above described problem as well (I've added it to the affected versions)

> Memory Leak with abnormal disconnecting consumers
> -------------------------------------------------
>
>                 Key: AMQ-6094
>                 URL: https://issues.apache.org/jira/browse/AMQ-6094
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.13.0, 5.12.2
>         Environment: Unmodified release version of ActiveMQ 5.13.0 (http://www.eu.apache.org/dist/activemq/5.13.0/apache-activemq-5.13.0-bin.tar.gz)
>            Reporter: Dominik Bieringer
>         Attachments: Test.java
>
>
> On our production ActiveMQ broker (processes around 10 000 messages / sec in average)
we have encountered situations where queues started blocking completely after running without
problems for a couple of days.
> When taking a look at the activemq logs, we can see messages like this (I've changed
queue names and client IPs):
> 2015-12-17 20:52:37,375 | INFO  | Usage(default:memory:queue://Consumer.AAA.VirtualTopic.OFFER:memory)
percentUsage=100%, usage=104858538, limit=104857600, percentUsageMinDelta=1%;Parent:Usage(default:memory)
percentUsage=42%, usage=305669289, limit=720424141, percentUsageMinDelta=1%: Usage Manager
Memory Limit reached. Producer (ID:ip-172-30-0-97-38230-1450370654525-1:8:1:1) stopped to
prevent flooding queue://Consumer.AAA.VirtualTopic.OFFER. See http://activemq.apache.org/producer-flow-control.html
for more info (blocking for: 151s) | org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///1.1.1.1:36128@61616
> The strange thing is, when taking a look at the admin interface, there are no messages
queued in the above mentioned queue and also purging the queue does not help.
> The only thing that works (that we found out so far) (to get the broker process messages
again) in the above situation is:
>  * delete the queue (it then is recreated automatically and works again)
>  * restart the broker
> I have now tried to reproduce the situation locally and come up with a test case that,
while I am not sure if that is the exact problem that we face in production, at least produces
the same problem as mentioned above. I have noticed that we sometimes have network issues
between the clients and the broker and therefore have done something similar in the test code.
> The test code launches 4 producing threads and 4 consuming threads. The producers >
1000 messages / sec to the queues and the consumers just read them. Once after a while (every
10 seconds), one of the consuming threads is interrupted and then, with a delay of another
10 seconds, the connection is cleaned up (to free up the allocated messages that are already
in the dead connections prefetchBuffer). 
> When running the test case on a fresh download of activeMQ 5.13.0, it takes a long time
until the broker completey blocks, as it takes time for the memory to fill up. However, when
checking JMX stats, it is clearly visible, that the following metrics behave strangely:
>  * CursorMemoryUsage
>  * MemoryUsageByteCount
> Both above metrics are quite constant for some time, and then, once a thread gets interrupted
and the connection cleaned up, it suddenly increases by couple of mbytes ... then, again,
while the consumers and producers work normally, the size is quite constant, until again,
a consumer is interrupted, which again increases memory for couple of mbytes ... and this
continues until memory is completely full and no messages can pass anymore through the broker.
> To speed things up, a lower memory limit can be placed on the queue in the activemq.xml
configuration file, which will lead to shorter waiting time before the broker blocks messages
on the queue.
> Even terminating the client jvm does not free up resources on the broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message