activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Broker Leak
Date Thu, 13 Dec 2012 01:30:16 GMT
Bad choice of words.. not "destined" but could run out of memory if
consumers cannot keep up with producers


On Wed, Dec 12, 2012 at 5:51 PM, Christian Posta
<christian.posta@gmail.com>wrote:

> So i guess in general i would think that a configuration with no PFC, vm
> cursors, and no persistence is destined to run out of memory.
>
> I wonder if what's happening in your case is these temporary queues are
> getting filled with messages that aren't being consumed and then when the
> reference hangs around in the Producer exchange, it's making a dent in the
> heap space. If this happens enough times it seems like it can OOM. I'll try
> to reproduce that on my side.
>
>
> On Wed, Dec 12, 2012 at 7:55 AM, Jerry Cwiklik <cwiklik@us.ibm.com> wrote:
>
>> Christian, thanks. The most recent experiment was run with a much smaller
>> broker, Xmx=360M. I am trying to reproduce the problem I am seeing in a
>> larger system which is not easy to debug. I can't share the large heap
>> dump
>> as it may contain proprietary data. But thanks for the offer to help
>> examine
>> it.
>>
>> I know that the large heap dump looks exactly as the screenshot I attached
>> to the initial post. It looks to me that somehow messages in temp queues
>> are
>> retained in broker memory. I watched this broker OOM and I can say for
>> sure
>> that none of the live temp queues showed any build up before the OOM.
>>
>> Assuming for a moment that this problem is induced by an application, what
>> could possibly be causing this build up of ProducerBrokerExchange objects
>> which hold references to messages in temp queues? Again, I know that there
>> is no buildup in live temp queues so this is not the case of a slow
>> consumer
>> not keeping up with processing msg from a temp queue.
>>
>> As mentioned in the original post, we run with producerFlowControl=false,
>> no
>> persistence, and vm cursor. I ran netstat on a machine where the broker is
>> running and dont see any FIN_WAIT nor WAIT_CLOSE. There are no messages in
>> brokers log saying that memory usage is high. In fact, the jconsole broker
>> MBean doesnt show it being short on memory. The memory usage shows as 0!
>> Yet, the brokers hits OOM.
>>
>> -Jerry C
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660538.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message