activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Frugia, Kirby A" <kirby.fru...@hp.com>
Subject RE: Broker not releasing memory
Date Mon, 08 Dec 2008 14:28:24 GMT
I can reproduce this with the non-embedded, out-of-the-box broker.  I also think it's an under-the-hood
problem.

Until we get a fix, we've set pending message queue limits on our busy topics.  So, for example,
the broker will only queue up 500 messages on a particular topic per consumer.  This will
prevent things from getting too out of hand on a single client crash, but is not the solution
to the problem.

-Kirby

-----Original Message-----
From: Will van der Leij [mailto:will@stonethree.com]
Sent: Monday, December 08, 2008 12:57 AM
To: users@activemq.apache.org
Subject: Re: Broker not releasing memory


I use an embedded broker too and have gone through all the possible
iterations of config options for threading, acknowledgements and message
storage.

Alas, an apparent "memory leak" still persists with slow consumers that
close/die with messages still in-flight/pending. I've had several other
people report the same problem with different clients leading me to believe
that there must be something odd under the hood.


Rick Blair wrote:
>
> Hi Will,
>
> We use an embedded broker  Here is the url.
>
> tcp://{0}:{1}?jms.useAsyncSend=false&jms.optimizeAcknowledge=false&jms.prefe
> tchPolicy.all=1000";
>
>  The problem started when optimizeAcknowledge=true was used.  Also I cannt
> get old messages in the topics to timeout.  Some do, most don¹t.
>
>
> Hope this helps.
>
> Rick
> --
> Tìoraidh!
>
> Rick Blair
> Associate Technical Fellow
> Boeing Phantom Works
> Network Systems Technology
> Information Management Program
> M/S:  42-50
> Voice:  (206) 544-1610
>
>
>
>> From: Will van der Leij <will@stonethree.com>
>> Reply-To: <users@activemq.apache.org>
>> Date: Wed, 3 Dec 2008 23:32:31 -0800 (PST)
>> To: <users@activemq.apache.org>
>> Subject: Re: Broker not releasing memory
>>
>>
>> Hi Rick,
>> As I understand optimizeAcknowledge is disabled by default.
>> Unfortunately,
>> setting it explicitely has not had an affect on our memory usage.
>> Could you perhaps briefly describe the rest of your configuration? I'm
>> presuming that it is a combination of options that would do the trick.
>>
>> Regards,
>> Will
>>
>>
>>
>> Rick Blair wrote:
>>>
>>> Hi Kirby,
>>>
>>> We have seen exactly the same behaviour.  As a work around we set
>>> optimizeAcknowdlege to false.  I did not run down the root cause,
>>> however.
>>>
>>> --
>>> Cheers!
>>>
>>> Rick Blair
>>>
>>>
>>>> From: "Frugia, Kirby A" <kirby.frugia@hp.com>
>>>> Reply-To: <users@activemq.apache.org>
>>>> Date: Wed, 3 Dec 2008 15:43:30 +0000
>>>> To: "users@activemq.apache.org" <users@activemq.apache.org>
>>>> Conversation: Broker not releasing memory
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>> This is fairly easy to reproduce if you create one app which sends
>>>> messages
>>>> quickly and another which consumes the messages and sleeps before
>>>> ack'ing.
>>>> Once you see broker memory usage going up (through activemq-admin),
>>>> kill
>>>> the
>>>> consumer application.  Then run activemq-admin again some time later
>>>> and
>>>> you'll see the same memorypercentusage.
>>>>
>>>> Or, let the consumer run the broker out of memory and you'll see the
>>>> producer
>>>> get blocked.  Kill both apps and you'll never see memory go back down
>>>> again
>>>> and you'll never be able to send messages again.
>>>>
>>>> Thanks,
>>>> Kirby
>>>>
>>>> -----Original Message-----
>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>> Sent: Wednesday, December 03, 2008 12:17 AM
>>>> To: users@activemq.apache.org
>>>> Subject: RE: Broker not releasing memory
>>>>
>>>>
>>>> Well, I'm reticent to suggest something going on under the hood without
>>>> taking a good look at it, however, if there is something wrong in the
>>>> broker
>>>> then I presume it has to do with a "sweeper"-like thread that
>>>> periodically
>>>> cleans up queues and topics that are left pending and consumerless
>>>> (similar
>>>> to how sweeper threads would clean up expired messages etc.
>>>>
>>>> Then again, we are the only two that I know of describing this
>>>> behaviour.
>>>> This suggests that either there is something bizarrely odd about our
>>>> configurations or we are, indeed, correct :)
>>>>
>>>>
>>>>
>>>> Frugia, Kirby A wrote:
>>>>>
>>>>> Yes, this is exactly what we are seeing.  Any ideas?
>>>>>
>>>>> -----Original Message-----
>>>>> From: Will van der Leij [mailto:will@stonethree.com]
>>>>> Sent: Thursday, November 27, 2008 11:42 PM
>>>>> To: users@activemq.apache.org
>>>>> Subject: Re: Broker not releasing memory
>>>>>
>>>>>
>>>>> We see similar behaviour in a slightly different setup so I'd like to
>>>>> elaborate a bit in the hope of finding out if this is either intended
>>>>> behaviour or a memory leak.
>>>>>
>>>>> Essentially, with a single or multiple producers and a single or
>>>>> multiple
>>>>> consumers on a single topic:
>>>>>  - We send messages faster than the consumer can read asynchronously
>>>>>  - When the send queue is larger than the consumer's prefetch buffer
>>>>> then
>>>>>    we correctly see messages filling up the broker memory
>>>>> (memoryPercentUSage in JMX)
>>>>>  - Killing the consumer (cleanly through a close or not-cleanly with
a
>>>>> terminate signal)
>>>>>    results in the memoryPercentUsage not being released
>>>>>    - even though there are no more consumers on the Topic
>>>>>    - this memory is never released until the broker is restarted
>>>>>
>>>>> I've tried it with persistence on and off, all the retroactive
>>>>> recovery
>>>>> options on and off, flow control on and off, caching on and off,
>>>>> asynchronous dispatching on and off, noLocal flag on and off etc.
>>>>>
>>>>> This is a very typical case of a consumer dying with inflight messages
>>>>> on
>>>>> the broker. Is the behaviour of the messages not being cleaned up or
>>>>> recovered intended (i.e. they are there for a reason) or is there
>>>>> soething
>>>>> wrnog with the setup.
>>>>>
>>>>> Many thanks
>>>>> Will van der Leij
>>>>>
>>>>>
>>>>>
>>>>> Frugia, Kirby A wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Sorry I dual-posted in the dev list.  I think I sent to the wrong
>>>>>> one...
>>>>>>
>>>>>> We are seeing an issue with our broker not releasing memory on
>>>>>> topics.
>>>>>>
>>>>>> Setup:
>>>>>> * Active MQ 5.1 (out of the box)
>>>>>> * Persistent messages sent by publishers
>>>>>> * Non-durable topics
>>>>>>
>>>>>> We were seeing an issue in production with slow consumers causing
the
>>>>>> broker to run out of memory.
>>>>>>
>>>>>> We wrote a test app, which has one publisher and one subscriber to
>>>>>> the
>>>>>> same topic.  The publisher sends messages on this topic frequently.
>>>>>> The
>>>>>> subscriber, upon receiving a message, goes to sleep for 10 minutes.
>>>>>>
>>>>>> Very quickly, the broker runs out of memory.  When this happens,
our
>>>>>> publishers can no longer send messages; the send is blocked. This
is
>>>>>> expected.  However, if we kill our application (which cleanly
>>>>>> disconnects
>>>>>> from the broker), the broker's memory usage (MemoryPercentUsage)
does
>>>>>> not
>>>>>> go back down.  Also, any new apps that startup will have their
>>>>>> publishers
>>>>>> blocked.
>>>>>>
>>>>>> Shouldn't the broker release the memory associated with these
>>>>>> messages?
>>>>>>
>>>>>> Thanks,
>>>>>> Kirby
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20730116.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20808585.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20828589.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>

--
View this message in context: http://www.nabble.com/Broker-not-releasing-memory-tp20662078p20890719.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message