activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Memoryleak when consuming messages from KahaDBStore
Date Thu, 26 Feb 2009 17:39:13 GMT
Jörg,
I finally did some analysis with yourkit today and indeed there is a
memory leak, I captured the issue and resolution with reference to
this thread in the issue:
https://issues.apache.org/activemq/browse/AMQ-2143

I think the resolution of the leak should help with more stable
enqueue rates. I will leave a steady state test running over night to
validate.

thanks for the analysis,
Gary.


2009/2/21 Cybexion <cybexion@email.de>:
>
> Hi Gary,
>
> we just made a new test which startet yesterday.
> JVM settings of AMQ where 3 GB heap size.
> We send about 800 msg / sec into the queue, the size was limited to 3 GB.
> During that time sending messages to the queue got slower and slower. Even
> after the queue was full and new messages have been thrown away. Futhermore,
> the memory consimption was increasing and increasing. I checked with
> visualvm and the heap was totally used. A full GC did not change anything.
> I tried to get a memdump with jmap but the jvm crashed :-(
> My current state is, that at least in my case AMQ is not usable for two
> reasons:
> 1. Obiously a mem leak
> 2. Sending messages to the queue takes more and more time the more messages
> have been send already.
>
> Best regards
>
> Jörg
>
>
>
>
> Gary Tully wrote:
>>
>> Hi Jörg,
>> I did some analysis of the per message allocations from your email. It
>> is clear that each store write allocates a bunch of objects, however,
>> from code inspection, I cannot see a point where they are leaked
>> through a reference.
>> There are two relevant points that may make static analysis a little
>> difficult. First is the fact that writes are asynchronous, so there
>> will be a small delay until write allocations are free for GC. This
>> should be very small in reality though.
>> Second is that store pages remain in memory until they are pushed out
>> of a LRU cache, a page has the possibility of referencing batch writes
>> and indexes with location information, ie: most of the per write
>> allocated objects.
>>
>> In short, we need to repeat your experiment with some more depth by
>> taking caching issues into consideration.  This is on my to do list.
>>
>> From a more high level view point, is it the case that a long running
>> test, with say 2000m/s in and out of the broker will eventually run
>> out of memory?
>>
>> I ask this because there are two sequential questions: 1) is there
>> really a memory leak (long running test gets into trouble) 2) if there
>> is a leak, where is it?
>>
>> Gary.
>>
>> 2009/2/13 Cybexion <cybexion@email.de>:
>>>
>>> Hello,
>>>
>>> I just tried something with the Netbeans memory profiler.
>>> My assumption is:
>>> If nothing hass been added to an application then after 2 full runs of
>>> the
>>> Garbace Collector the memory should be the same.
>>> In the past we have seen that memory was increasing while messages have
>>> been
>>> processed by AMQ. For me that means that something stays in memory even
>>> after those messages have been dequeued.
>>>
>>> My test looks like this:
>>> 1. start a fresh AMQ instance, send some messages to a queue for
>>> initialisation
>>> 2. do a full GC
>>> 3. Take a memory snapshot
>>> 4. send exactly 1000 message to the queue
>>> 5. consume all 1000 messages from the queue
>>> 6. do a full GC
>>> 7. Take a another memory snapshot
>>>
>>> Assumption:
>>> I would expect that no references to messages do exist any more in the
>>> AMQ
>>> JVM because they have all been processed. The memory should look more or
>>> less like it was before the test.
>>> Now I did a memory comparision of my two snapshots.
>>> The result looks like this:
>>> http://www.nabble.com/file/p22003872/AMQClasses.jpg
>>> From within the "org.apache.activemq" package there are exactly 1000
>>> MessageKeys, 1000 RemoveMessageCommands and 1000 AdMessageCommands left
>>> over.
>>> Shouldn't those have disappeared because the queue is empty???
>>>
>>> Additionaly, if we look at the "org.apache.kaha" package:
>>> http://www.nabble.com/file/p22003872/KahaClasses.jpg
>>> There are 2000 DataByteArrayOutputstreams, 2000 Location objects and 2000
>>> WriteCommands still inside the memory. I would also have expected that
>>> those
>>> do not exist after the queue is empty.
>>>
>>> Well, personally this does not help me, but maybe one of the developers
>>> can
>>> convert this information into a possible bug inside the code?
>>>
>>> Help would really be appreciated.
>>>
>>> Thanks
>>>
>>> Jörg
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22003872.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Memoryleak-when-consuming-messages-from-KahaDBStore-tp21991929p22141262.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>



-- 
http://blog.garytully.com

Open Source SOA
http://FUSESource.com

Mime
View raw message