activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Problems with prefetch and TemporaryQueues
Date Thu, 14 Jan 2010 17:51:40 GMT
Great that you have a solution that works.
I think you have a good point here. I cannot think of a reason to have auto
create for temp queues save that it negates the need to call
session.createTempX which is not well behaved w.r.t JMS semantics. Can you
open a jira issue to track this?

As to the Junit test case, really all you need to do is follow some of the
existing test cases in activemq-core/src/test - look for something with
*Temp*Test in the file name and you should get close to what you need.

2010/1/14 Zemus <simon.nilsson@tieto.com>

>
> Thanks for the answer, Gary.
>
> I rebuilt ActiveMQ with
> setAutoCreateDestinations(false);
> in the constructor of TempQueueRegion.
> It seems to solve the problem. I put some logging into
> AbstractRegion.lookup
> to verify this.
>
> Is there any reason to have autocreating enabled by default for TempQueues?
> Since the creator of the queue is the only that can consume from it, that
> would mean that the sender creating the queue has the only connection that
> can read from it. Or am I missing something?
>
> Perhaps the default value should be changed to false if the common usage of
> TempQueue benefits from it?
>
> As for a JUnit test case I will have to ask my employers if I can spend
> time
> on that. Are there any ActiveMQ specific guidelines for how they should be
> constructed?
>
> Best Regards
> Zemus
>
>
>
> Gary Tully wrote:
> >
> > this sounds like a reasonable theory.
> > An ActiveMQConnection by default registers interest in the advisory
> > messages
> > for temp destination removal so it does try and track temp queues as you
> > suggest.
> > So if advisory support is enabled for the broker the window for
> recreation
> > of a temp queue should be small.
> >
> > However I think there could be a window between temp queue deletion and
> > the
> > advisory message dispatch such that the replying connection (what you
> deem
> > server) could consider the queue valid when it is not, resulting in a new
> > temp queue auto creation.
> >
> > Can you make a junit test case from what you have and attach it to a new
> > jira issue.
> >
> > I see two solutions, have the broker optionally not auto create temp
> > destinations or have it send out the advisory messages for temp
> > destination
> > deletion before the deletion happens. (this will still depend on the
> > consumer in a connection reacting to the advisory though which will still
> > leave a window when a connection can think the temp queue is valid - it
> > will
> > help but may not be sufficient.)
> >
> > Exposing an option to not auto create temp destinations may be the
> > simplest
> > solution. Such an option is currently on the
> >
> org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
> > if you control the broker start you may want to explore disabling this
> > through code. Not sure if all that you need is there in the api at the
> > moment, but it could provide a workaround that is worth investigating a
> > bit.
> >
> > 2010/1/13 Zemus <simon.nilsson@tieto.com>
> >
> >>
> >>
> >> Zemus wrote:
> >> >
> >> > Hi,
> >> >
> >> > I have a problem where TemporaryQueues are left (with 0 consumers)
> >> after
> >> > the applications creating them have finished.
> >> >
> >> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6
> >> GHz,
> >> 3
> >> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
> >> >
> >> > I've tried to recreate a simpler version of the scenario. Basically I
> >> have
> >> > a server that creates a Topic where clients that wants to subscribe
> may
> >> > post a message (with a TemporaryQueue as the JMS reply) in order to be
> >> > added. The server then has a thread that sends messages to all the
> >> > TemporaryQueues. (I know a simple Topic could push the updates here,
> >> but
> >> > in the real system the clients can receive information only intended
> >> for
> >> > one single client.)
> >> >
> >> > In my scenario the server is faster at generating messages than the
> >> > clients are in processing them so the queues start to build up. Pretty
> >> > soon (if clients are added and removed once a minute or so) it will
> hit
> >> a
> >> > limit and the processing of messages in the broker seems to come to a
> >> > halt. I think it is when the prefetch limit is reached.
> >> > The real problem here is that if a client disconnects successfully
> >> (might
> >> > hang in the broker communication) it is possible for the server to
> >> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
> >> > messages to it. This way the TemporaryQueue is left on the broker with
> >> no
> >> > consumer (the client disconnected) with its queued up messages
> counting
> >> > towards the prefetch limit.
> >> >
> >> > By running JConsole and purging the queue I can get the broker to wake
> >> up,
> >> > but if many messages are queued up the consumerless TempQueue will
> >> queue
> >> > those messages which might again cause the prefetch limit to be
> reached
> >> > and halt the broker.
> >> > Also, I've seen old TemporaryQueues that were removed minutes ago come
> >> > back and start queueing buffered messages when I start purging other
> >> > consumerless TempQueues.
> >> >
> >> >
> >> > In short I think there are two problems here.
> >> > First, the detection of the client disconnection is not instant on the
> >> > server. This makes it possible to send messages to the TemporaryQueue
> >> > without getting an exception after it should have been removed.
> >> >
> >> > The second is that the TemporaryQueue is able to resurrect after its
> >> > creator/consumer has left. In other words it has outlived its
> >> connection.
> >> >
> >> >
> >> > I attach a simple client and server that are able to reproduce this
> >> > reliably (at least on my computer).
> >> > Just run the server and maybe two clients. Restart the clients perhaps
> >> 1-2
> >> > times per minute and the issue will show pretty quickly. At least on
> my
> >> > P4, maybe a computer with multiple cores will behave different.
> >> >
> >> > Regards
> >> > Zemus
> >> >
> >> >  http://old.nabble.com/file/p27130529/TestClient.java TestClient.java
> >> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
> >> > ClientShutdownHook.java
> >> >
> >> >  http://old.nabble.com/file/p27130529/TestServer.java TestServer.java
> >> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
> >> > ServerShutdownHook.java
> >> >
> >>
> >>
> >> After more investigation I've changed my mind about the nature of the
> >> problem. The prefetch limit is not involved.
> >>
> >> My new theory goes like this:
> >>
> >> 1. The temporary queue is removed correctly when the client exists and
> >> closes its connection.
> >>
> >> 2. The server might receive an exception when it tries to send to the
> >> removed temporary queue. BUT sometimes it doesn't get that exception
> >> (perhaps because the broker was soo busy that it didn't have time to
> >> notify
> >> the activemq part of the server in time) and instead the broker creates
> a
> >> new queue with the same name, although without any consumer, when the
> >> server
> >> sends the next message to the stored destination.
> >>
> >> 3. The server knows nothing about this and keeps sending messages that
> >> pile
> >> up in the broker, which in turn, eventually runs out of memory (at least
> >> jconsole reports MemoryPercentageUsed 100 for the broker object).
> >>
> >>
> >> If this sounds reasonable, my questions are instead:
> >>
> >> * How can I prevent the broker from creating a new temporary queue
> >> because
> >> of the server's late message?
> >>
> >> * Would listening to advisory support for the temp queue at the server
> >> side
> >> notify me in time to stop the sending of any more messages when the temp
> >> queue is removed?
> >>
> >> * Or is there a better way that would stop the server from sending a
> >> message
> >> to a removed temp queue?
> >>
> >> Regards
> >> Zemus
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27162131.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message