activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Broker Leak
Date Fri, 14 Dec 2012 17:44:50 GMT
Jerry,

Thanks again for demonstrating this.
It's been fixed in the trunk.

https://issues.apache.org/jira/browse/AMQ-4222

Thanks,
Christian


On Thu, Dec 13, 2012 at 11:17 AM, Christian Posta <christian.posta@gmail.com
> wrote:

> Awesome, thanks for coming back and explaining what happened there. I was
> super curious about this one.
> I still think there's room for a unit test to document this and show that
> the reference in the ProducerBrokerExchange still holds onto the "region"
> destinations. This reference could be cleared at the end of a send since
> it's never used again and would allow the broker to reclaim the destination
> completely.
>
> I'll write the test for that and change that reference!!
> Tracked here:
>
> https://issues.apache.org/jira/browse/AMQ-4222
>
>
>
> On Thu, Dec 13, 2012 at 10:17 AM, Jerry Cwiklik <cwiklik@us.ibm.com>wrote:
>
>> Christian, after much pain and suffering I finally figured out what is
>> going
>> on. Our system is quite complicated and involves many producers that send
>> large messages (600K-1.5M) to a relatively few multi-threaded consumers
>> (services) which run "forever". The producers are transient and can be
>> killed by our custom job scheduler at any time via kill -9 to make room
>> for
>> other producers. We run the broker with 10G heap.
>>
>> The consumer is coded to group and cache Sessions with a Connection which
>> has an inactivity timer  associated with it. Every time a message is sent,
>> the timer is restarted. If the timer pops (default 30minutes), the
>> Sessions,
>> MessageProducers and a Connection are closed due to inactivity.
>>
>> This worked perfectly fine until about 4 weeks ago when we started
>> experiencing broker OOM problem. While the broker was running we could
>> see a
>> steady (fast) rise in the heap usage in a jConsole. After a couple of days
>> the broker's jvm would OOM.
>>
>> The problem started happening when we introduced pingers for the
>> Consumers.
>> Every minute a pinger sends a message to a Consumer to make sure its
>> alive.
>> The Consumer replies to the pinger request and restarts inactivity timer.
>> It
>> took me awhile to see the bug in our application, but eventually I
>> determined that our timer behaves incorrectly as it is associated with a
>> Connection not individual Sessions. The Sessions go stale due to producer
>> getting killed, and any messages in the broker referenced by
>> ProducerExchange object are retained indefinitely causing a leak in the
>> broker. As you explained it to me, the broker uses lazy approach to
>> cleanup.
>> Meaning it cleans up on a new message from the Producer. In our case, the
>> Producer never sends anything and thus no cleanup is ever done.
>>
>> The fix for this is to create a timestamp for every Session when it was
>> last
>> used to message to the broker. At fixed intervals a Session Reaper thread
>> wakes up and checks the timestamp of every Session to determine if it has
>> been inactive for a max allowed time and if so, to close it.
>>
>> So the problem was caused by an application bug and the fact that the
>> broker
>> takes a lazy approach to cleanup. As a side note, under the described
>> scenario, I've noticed that the broker memory usage (shown in jConsole)
>> indicated 0 even though there were ton of messages in the heap with valid
>> references (held by ProducerExchange).
>>
>> Thanks Christian for your help
>>
>> -Jerry C
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660618.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>


-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message