Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <28698719.post@talk.nabble.com>
Date: Thu, 27 May 2010 12:19:53 -0700 (PDT)
From: odysseyfx <info@bradwillard.com>
To: users@activemq.apache.org
Subject: Re: Topic Memory Increases until broker dies
In-Reply-To: <28370484.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <28370484.post@talk.nabble.com>


I've seen this problem fairly frequently.  This is caused by  a message
producer on a topic creating messages faster than a consumer can consume
them.  Those messages then get backed up on the broker internal dispatch
queue for that session until all resources are consumed when the broker
deadlocks.

There are a few solutions to fix this problem that I have found: configuring
and enabling flow control (which will throttle the message producer), enable
a message eviction policy (where the broker will start throwing out the
oldest messages of the lowest priority that aren't consumed fast enough), or
finally you can actually use a durable subscriber if the message producer
rate will slow down enough at some point in time for the consumers to catch
up, otherwise you'll wind up with the same problem backing up on a queue
instead.

Another solution is Virtual Topics, and activemq has a whole page on exactly
this scenario

http://activemq.apache.org/slow-consumer-handling.html
and
http://activemq.apache.org/virtual-destinations.html

I want to add that there is no tuning guide out there to get flow control to
work properly, and it's a pain in the ass to setup.  You'll need someone who
knows the code to help, there is no document on this but there should be.

Good luck,
Brad


E White wrote:
> 
> We have a java swing client that gets its data updates by subscribing to
> an ActiveMQ topic.  All clients (about a hundred at any given time)
> subscribe to the same topic.  Subscribers vary and are not durable, but
> the messages are persistent.
> 
> I have seen the "inflight count" slowly, but steadily increase and then
> the "memory percent used" value spike, leaving the broker deadlocked and
> no longer processing messages, but not failing.  The whole scenario takes
> about a week.  And this makes our client application look like it is hung
> after it occurs.
> 
> How do I keep messages from getting orphaned when a user closes their
> client?  I think that is where most of my "inflight" number is coming
> from.  Users close the client right as it has received a message but
> before it can send the ack. 
> How do I make "inflight" messages expire?  I've played around with setting
> "timetolive" in my test environment, but it does not seem to affect
> messages that are stuck "inflight".
> 
> We are using ActiveMQ 5.3.0.
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28698719.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.