activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <>
Subject Re: Memoryleak when consuming messages from KahaDBStore
Date Thu, 26 Feb 2009 17:39:13 GMT
I finally did some analysis with yourkit today and indeed there is a
memory leak, I captured the issue and resolution with reference to
this thread in the issue:

I think the resolution of the leak should help with more stable
enqueue rates. I will leave a steady state test running over night to

thanks for the analysis,

2009/2/21 Cybexion <>:
> Hi Gary,
> we just made a new test which startet yesterday.
> JVM settings of AMQ where 3 GB heap size.
> We send about 800 msg / sec into the queue, the size was limited to 3 GB.
> During that time sending messages to the queue got slower and slower. Even
> after the queue was full and new messages have been thrown away. Futhermore,
> the memory consimption was increasing and increasing. I checked with
> visualvm and the heap was totally used. A full GC did not change anything.
> I tried to get a memdump with jmap but the jvm crashed :-(
> My current state is, that at least in my case AMQ is not usable for two
> reasons:
> 1. Obiously a mem leak
> 2. Sending messages to the queue takes more and more time the more messages
> have been send already.
> Best regards
> Jörg
> Gary Tully wrote:
>> Hi Jörg,
>> I did some analysis of the per message allocations from your email. It
>> is clear that each store write allocates a bunch of objects, however,
>> from code inspection, I cannot see a point where they are leaked
>> through a reference.
>> There are two relevant points that may make static analysis a little
>> difficult. First is the fact that writes are asynchronous, so there
>> will be a small delay until write allocations are free for GC. This
>> should be very small in reality though.
>> Second is that store pages remain in memory until they are pushed out
>> of a LRU cache, a page has the possibility of referencing batch writes
>> and indexes with location information, ie: most of the per write
>> allocated objects.
>> In short, we need to repeat your experiment with some more depth by
>> taking caching issues into consideration.  This is on my to do list.
>> From a more high level view point, is it the case that a long running
>> test, with say 2000m/s in and out of the broker will eventually run
>> out of memory?
>> I ask this because there are two sequential questions: 1) is there
>> really a memory leak (long running test gets into trouble) 2) if there
>> is a leak, where is it?
>> Gary.
>> 2009/2/13 Cybexion <>:
>>> Hello,
>>> I just tried something with the Netbeans memory profiler.
>>> My assumption is:
>>> If nothing hass been added to an application then after 2 full runs of
>>> the
>>> Garbace Collector the memory should be the same.
>>> In the past we have seen that memory was increasing while messages have
>>> been
>>> processed by AMQ. For me that means that something stays in memory even
>>> after those messages have been dequeued.
>>> My test looks like this:
>>> 1. start a fresh AMQ instance, send some messages to a queue for
>>> initialisation
>>> 2. do a full GC
>>> 3. Take a memory snapshot
>>> 4. send exactly 1000 message to the queue
>>> 5. consume all 1000 messages from the queue
>>> 6. do a full GC
>>> 7. Take a another memory snapshot
>>> Assumption:
>>> I would expect that no references to messages do exist any more in the
>>> AMQ
>>> JVM because they have all been processed. The memory should look more or
>>> less like it was before the test.
>>> Now I did a memory comparision of my two snapshots.
>>> The result looks like this:
>>> From within the "org.apache.activemq" package there are exactly 1000
>>> MessageKeys, 1000 RemoveMessageCommands and 1000 AdMessageCommands left
>>> over.
>>> Shouldn't those have disappeared because the queue is empty???
>>> Additionaly, if we look at the "org.apache.kaha" package:
>>> There are 2000 DataByteArrayOutputstreams, 2000 Location objects and 2000
>>> WriteCommands still inside the memory. I would also have expected that
>>> those
>>> do not exist after the queue is empty.
>>> Well, personally this does not help me, but maybe one of the developers
>>> can
>>> convert this information into a possible bug inside the code?
>>> Help would really be appreciated.
>>> Thanks
>>> Jörg
>>> --
>>> View this message in context:
>>> Sent from the ActiveMQ - User mailing list archive at
>> --
>> Open Source SOA
> --
> View this message in context:
> Sent from the ActiveMQ - User mailing list archive at


Open Source SOA

View raw message