activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Broker Leak
Date Thu, 13 Dec 2012 18:17:10 GMT
Awesome, thanks for coming back and explaining what happened there. I was
super curious about this one.
I still think there's room for a unit test to document this and show that
the reference in the ProducerBrokerExchange still holds onto the "region"
destinations. This reference could be cleared at the end of a send since
it's never used again and would allow the broker to reclaim the destination
completely.

I'll write the test for that and change that reference!!
Tracked here:

https://issues.apache.org/jira/browse/AMQ-4222



On Thu, Dec 13, 2012 at 10:17 AM, Jerry Cwiklik <cwiklik@us.ibm.com> wrote:

> Christian, after much pain and suffering I finally figured out what is
> going
> on. Our system is quite complicated and involves many producers that send
> large messages (600K-1.5M) to a relatively few multi-threaded consumers
> (services) which run "forever". The producers are transient and can be
> killed by our custom job scheduler at any time via kill -9 to make room for
> other producers. We run the broker with 10G heap.
>
> The consumer is coded to group and cache Sessions with a Connection which
> has an inactivity timer  associated with it. Every time a message is sent,
> the timer is restarted. If the timer pops (default 30minutes), the
> Sessions,
> MessageProducers and a Connection are closed due to inactivity.
>
> This worked perfectly fine until about 4 weeks ago when we started
> experiencing broker OOM problem. While the broker was running we could see
> a
> steady (fast) rise in the heap usage in a jConsole. After a couple of days
> the broker's jvm would OOM.
>
> The problem started happening when we introduced pingers for the Consumers.
> Every minute a pinger sends a message to a Consumer to make sure its alive.
> The Consumer replies to the pinger request and restarts inactivity timer.
> It
> took me awhile to see the bug in our application, but eventually I
> determined that our timer behaves incorrectly as it is associated with a
> Connection not individual Sessions. The Sessions go stale due to producer
> getting killed, and any messages in the broker referenced by
> ProducerExchange object are retained indefinitely causing a leak in the
> broker. As you explained it to me, the broker uses lazy approach to
> cleanup.
> Meaning it cleans up on a new message from the Producer. In our case, the
> Producer never sends anything and thus no cleanup is ever done.
>
> The fix for this is to create a timestamp for every Session when it was
> last
> used to message to the broker. At fixed intervals a Session Reaper thread
> wakes up and checks the timestamp of every Session to determine if it has
> been inactive for a max allowed time and if so, to close it.
>
> So the problem was caused by an application bug and the fact that the
> broker
> takes a lazy approach to cleanup. As a side note, under the described
> scenario, I've noticed that the broker memory usage (shown in jConsole)
> indicated 0 even though there were ton of messages in the heap with valid
> references (held by ProducerExchange).
>
> Thanks Christian for your help
>
> -Jerry C
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-Leak-tp4660437p4660618.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message