activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Frugia, Kirby A" <kirby.fru...@hp.com>
Subject RE: Broker not releasing memory
Date Wed, 03 Dec 2008 15:43:30 GMT
This is fairly easy to reproduce if you create one app which sends messages quickly and another
which consumes the messages and sleeps before ack'ing.  Once you see broker memory usage going
up (through activemq-admin), kill the consumer application.  Then run activemq-admin again
some time later and you'll see the same memorypercentusage.

Or, let the consumer run the broker out of memory and you'll see the producer get blocked.
 Kill both apps and you'll never see memory go back down again and you'll never be able to
send messages again.

Thanks,
Kirby

-----Original Message-----
From: Will van der Leij [mailto:will@stonethree.com]
Sent: Wednesday, December 03, 2008 12:17 AM
To: users@activemq.apache.org
Subject: RE: Broker not releasing memory


Well, I'm reticent to suggest something going on under the hood without
taking a good look at it, however, if there is something wrong in the broker
then I presume it has to do with a "sweeper"-like thread that periodically
cleans up queues and topics that are left pending and consumerless (similar
to how sweeper threads would clean up expired messages etc.

Then again, we are the only two that I know of describing this behaviour.
This suggests that either there is something bizarrely odd about our
configurations or we are, indeed, correct :)



Frugia, Kirby A wrote:
>
> Yes, this is exactly what we are seeing.  Any ideas?
>
> -----Original Message-----
> From: Will van der Leij [mailto:will@stonethree.com]
> Sent: Thursday, November 27, 2008 11:42 PM
> To: users@activemq.apache.org
> Subject: Re: Broker not releasing memory
>
>
> We see similar behaviour in a slightly different setup so I'd like to
> elaborate a bit in the hope of finding out if this is either intended
> behaviour or a memory leak.
>
> Essentially, with a single or multiple producers and a single or multiple
> consumers on a single topic:
>  - We send messages faster than the consumer can read asynchronously
>  - When the send queue is larger than the consumer's prefetch buffer then
>    we correctly see messages filling up the broker memory
> (memoryPercentUSage in JMX)
>  - Killing the consumer (cleanly through a close or not-cleanly with a
> terminate signal)
>    results in the memoryPercentUsage not being released
>    - even though there are no more consumers on the Topic
>    - this memory is never released until the broker is restarted
>
> I've tried it with persistence on and off, all the retroactive recovery
> options on and off, flow control on and off, caching on and off,
> asynchronous dispatching on and off, noLocal flag on and off etc.
>
> This is a very typical case of a consumer dying with inflight messages on
> the broker. Is the behaviour of the messages not being cleaned up or
> recovered intended (i.e. they are there for a reason) or is there soething
> wrnog with the setup.
>
> Many thanks
> Will van der Leij
>
>
>
> Frugia, Kirby A wrote:
>>
>> Hi All,
>>
>> Sorry I dual-posted in the dev list.  I think I sent to the wrong one...
>>
>> We are seeing an issue with our broker not releasing memory on topics.
>>
>> Setup:
>> * Active MQ 5.1 (out of the box)
>> * Persistent messages sent by publishers
>> * Non-durable topics
>>
>> We were seeing an issue in production with slow consumers causing the
>> broker to run out of memory.
>>
>> We wrote a test app, which has one publisher and one subscriber to the
>> same topic.  The publisher sends messages on this topic frequently.  The
>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>
>> Very quickly, the broker runs out of memory.  When this happens, our
>> publishers can no longer send messages; the send is blocked. This is
>> expected.  However, if we kill our application (which cleanly disconnects
>> from the broker), the broker's memory usage (MemoryPercentUsage) does not
>> go back down.  Also, any new apps that startup will have their publishers
>> blocked.
>>
>> Shouldn't the broker release the memory associated with these messages?
>>
>> Thanks,
>> Kirby
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message