activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Problems with prefetch and TemporaryQueues
Date Wed, 13 Jan 2010 16:17:07 GMT
this sounds like a reasonable theory.
An ActiveMQConnection by default registers interest in the advisory messages
for temp destination removal so it does try and track temp queues as you
suggest.
So if advisory support is enabled for the broker the window for recreation
of a temp queue should be small.

However I think there could be a window between temp queue deletion and the
advisory message dispatch such that the replying connection (what you deem
server) could consider the queue valid when it is not, resulting in a new
temp queue auto creation.

Can you make a junit test case from what you have and attach it to a new
jira issue.

I see two solutions, have the broker optionally not auto create temp
destinations or have it send out the advisory messages for temp destination
deletion before the deletion happens. (this will still depend on the
consumer in a connection reacting to the advisory though which will still
leave a window when a connection can think the temp queue is valid - it will
help but may not be sufficient.)

Exposing an option to not auto create temp destinations may be the simplest
solution. Such an option is currently on the
org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
if you control the broker start you may want to explore disabling this
through code. Not sure if all that you need is there in the api at the
moment, but it could provide a workaround that is worth investigating a bit.

2010/1/13 Zemus <simon.nilsson@tieto.com>

>
>
> Zemus wrote:
> >
> > Hi,
> >
> > I have a problem where TemporaryQueues are left (with 0 consumers) after
> > the applications creating them have finished.
> >
> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6 GHz,
> 3
> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
> >
> > I've tried to recreate a simpler version of the scenario. Basically I
> have
> > a server that creates a Topic where clients that wants to subscribe may
> > post a message (with a TemporaryQueue as the JMS reply) in order to be
> > added. The server then has a thread that sends messages to all the
> > TemporaryQueues. (I know a simple Topic could push the updates here, but
> > in the real system the clients can receive information only intended for
> > one single client.)
> >
> > In my scenario the server is faster at generating messages than the
> > clients are in processing them so the queues start to build up. Pretty
> > soon (if clients are added and removed once a minute or so) it will hit a
> > limit and the processing of messages in the broker seems to come to a
> > halt. I think it is when the prefetch limit is reached.
> > The real problem here is that if a client disconnects successfully (might
> > hang in the broker communication) it is possible for the server to
> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
> > messages to it. This way the TemporaryQueue is left on the broker with no
> > consumer (the client disconnected) with its queued up messages counting
> > towards the prefetch limit.
> >
> > By running JConsole and purging the queue I can get the broker to wake
> up,
> > but if many messages are queued up the consumerless TempQueue will queue
> > those messages which might again cause the prefetch limit to be reached
> > and halt the broker.
> > Also, I've seen old TemporaryQueues that were removed minutes ago come
> > back and start queueing buffered messages when I start purging other
> > consumerless TempQueues.
> >
> >
> > In short I think there are two problems here.
> > First, the detection of the client disconnection is not instant on the
> > server. This makes it possible to send messages to the TemporaryQueue
> > without getting an exception after it should have been removed.
> >
> > The second is that the TemporaryQueue is able to resurrect after its
> > creator/consumer has left. In other words it has outlived its connection.
> >
> >
> > I attach a simple client and server that are able to reproduce this
> > reliably (at least on my computer).
> > Just run the server and maybe two clients. Restart the clients perhaps
> 1-2
> > times per minute and the issue will show pretty quickly. At least on my
> > P4, maybe a computer with multiple cores will behave different.
> >
> > Regards
> > Zemus
> >
> >  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java
> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
> > ClientShutdownHook.java
> >
> >  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java
> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
> > ServerShutdownHook.java
> >
>
>
> After more investigation I've changed my mind about the nature of the
> problem. The prefetch limit is not involved.
>
> My new theory goes like this:
>
> 1. The temporary queue is removed correctly when the client exists and
> closes its connection.
>
> 2. The server might receive an exception when it tries to send to the
> removed temporary queue. BUT sometimes it doesn't get that exception
> (perhaps because the broker was soo busy that it didn't have time to notify
> the activemq part of the server in time) and instead the broker creates a
> new queue with the same name, although without any consumer, when the
> server
> sends the next message to the stored destination.
>
> 3. The server knows nothing about this and keeps sending messages that pile
> up in the broker, which in turn, eventually runs out of memory (at least
> jconsole reports MemoryPercentageUsed 100 for the broker object).
>
>
> If this sounds reasonable, my questions are instead:
>
> * How can I prevent the broker from creating a new temporary queue because
> of the server's late message?
>
> * Would listening to advisory support for the temp queue at the server side
> notify me in time to stop the sending of any more messages when the temp
> queue is removed?
>
> * Or is there a better way that would stop the server from sending a
> message
> to a removed temp queue?
>
> Regards
> Zemus
>
>
> --
> View this message in context:
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message