Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of gary.tully@gmail.com designates
 209.85.218.215 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DNobCExFn1y2T++olKGQIohHymbi/f/ptF25WN9j8lcD8v21c2ZRlXTXQZEewS9ZVr
         S2XzutlSQlEOvcTYvnrA0IEphDB40PqDZiiuxb4x5vpPXzcpIEExlskqmVqJK2kw32j8
         PYtmjp4qFBVMOnsR5uhXBgLaTrE5Spu4Km7EI=
MIME-Version: 1.0
In-Reply-To: <3a73c17c1001140951s473c53a6v3c8da155b100c1d2@mail.gmail.com>
References: <27130529.post@talk.nabble.com> <27147013.post@talk.nabble.com>
	 <3a73c17c1001130817p573c1999md6fbdc4010b250b2@mail.gmail.com>
	 <27162131.post@talk.nabble.com>
	 <3a73c17c1001140951s473c53a6v3c8da155b100c1d2@mail.gmail.com>
Date: Thu, 14 Jan 2010 18:09:00 +0000
Message-ID: <3a73c17c1001141009j7e2d0db4ibeba8c55bdfedd98@mail.gmail.com>
Subject: Re: Problems with prefetch and TemporaryQueues
From: Gary Tully <gary.tully@gmail.com>
To: users@activemq.apache.org
Content-Type: multipart/alternative; boundary=00032555b682137ec3047d23c8e7

--00032555b682137ec3047d23c8e7
Content-Type: text/plain; charset=ISO-8859-1

on second thoughts, the network of brokers support, which bridges temp
destinations by default, that needs (or builds on) auto create of temp
destinations. So the option to turn off auto creation for temp destinations
could be exposed as a configuration option on the broker but it can't be the
default behavior.

2010/1/14 Gary Tully <gary.tully@gmail.com>

> Great that you have a solution that works.
> I think you have a good point here. I cannot think of a reason to have auto
> create for temp queues save that it negates the need to call
> session.createTempX which is not well behaved w.r.t JMS semantics. Can you
> open a jira issue to track this?
>
> As to the Junit test case, really all you need to do is follow some of the
> existing test cases in activemq-core/src/test - look for something with
> *Temp*Test in the file name and you should get close to what you need.
>
> 2010/1/14 Zemus <simon.nilsson@tieto.com>
>
>
>> Thanks for the answer, Gary.
>>
>> I rebuilt ActiveMQ with
>> setAutoCreateDestinations(false);
>> in the constructor of TempQueueRegion.
>> It seems to solve the problem. I put some logging into
>> AbstractRegion.lookup
>> to verify this.
>>
>> Is there any reason to have autocreating enabled by default for
>> TempQueues?
>> Since the creator of the queue is the only that can consume from it, that
>> would mean that the sender creating the queue has the only connection that
>> can read from it. Or am I missing something?
>>
>> Perhaps the default value should be changed to false if the common usage
>> of
>> TempQueue benefits from it?
>>
>> As for a JUnit test case I will have to ask my employers if I can spend
>> time
>> on that. Are there any ActiveMQ specific guidelines for how they should be
>> constructed?
>>
>> Best Regards
>> Zemus
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > this sounds like a reasonable theory.
>> > An ActiveMQConnection by default registers interest in the advisory
>> > messages
>> > for temp destination removal so it does try and track temp queues as you
>> > suggest.
>> > So if advisory support is enabled for the broker the window for
>> recreation
>> > of a temp queue should be small.
>> >
>> > However I think there could be a window between temp queue deletion and
>> > the
>> > advisory message dispatch such that the replying connection (what you
>> deem
>> > server) could consider the queue valid when it is not, resulting in a
>> new
>> > temp queue auto creation.
>> >
>> > Can you make a junit test case from what you have and attach it to a new
>> > jira issue.
>> >
>> > I see two solutions, have the broker optionally not auto create temp
>> > destinations or have it send out the advisory messages for temp
>> > destination
>> > deletion before the deletion happens. (this will still depend on the
>> > consumer in a connection reacting to the advisory though which will
>> still
>> > leave a window when a connection can think the temp queue is valid - it
>> > will
>> > help but may not be sufficient.)
>> >
>> > Exposing an option to not auto create temp destinations may be the
>> > simplest
>> > solution. Such an option is currently on the
>> >
>> org.apache.activemq.broker.region.AbstractRegion.setAutoCreateDestinations(boolean)
>> > if you control the broker start you may want to explore disabling this
>> > through code. Not sure if all that you need is there in the api at the
>> > moment, but it could provide a workaround that is worth investigating a
>> > bit.
>> >
>> > 2010/1/13 Zemus <simon.nilsson@tieto.com>
>> >
>> >>
>> >>
>> >> Zemus wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I have a problem where TemporaryQueues are left (with 0 consumers)
>> >> after
>> >> > the applications creating them have finished.
>> >> >
>> >> > This scenario occurs for both ActiveMQ 4.1.1 and 5.3.0 on my P4 2.6
>> >> GHz,
>> >> 3
>> >> > GB RAM, Ubuntu 9.10, Sun JDK, default settings for ActiveMQ.
>> >> >
>> >> > I've tried to recreate a simpler version of the scenario. Basically I
>> >> have
>> >> > a server that creates a Topic where clients that wants to subscribe
>> may
>> >> > post a message (with a TemporaryQueue as the JMS reply) in order to
>> be
>> >> > added. The server then has a thread that sends messages to all the
>> >> > TemporaryQueues. (I know a simple Topic could push the updates here,
>> >> but
>> >> > in the real system the clients can receive information only intended
>> >> for
>> >> > one single client.)
>> >> >
>> >> > In my scenario the server is faster at generating messages than the
>> >> > clients are in processing them so the queues start to build up.
>> Pretty
>> >> > soon (if clients are added and removed once a minute or so) it will
>> hit
>> >> a
>> >> > limit and the processing of messages in the broker seems to come to a
>> >> > halt. I think it is when the prefetch limit is reached.
>> >> > The real problem here is that if a client disconnects successfully
>> >> (might
>> >> > hang in the broker communication) it is possible for the server to
>> >> > "resurrect" its TemporaryQueue, probably by sending prefetchbuffered
>> >> > messages to it. This way the TemporaryQueue is left on the broker
>> with
>> >> no
>> >> > consumer (the client disconnected) with its queued up messages
>> counting
>> >> > towards the prefetch limit.
>> >> >
>> >> > By running JConsole and purging the queue I can get the broker to
>> wake
>> >> up,
>> >> > but if many messages are queued up the consumerless TempQueue will
>> >> queue
>> >> > those messages which might again cause the prefetch limit to be
>> reached
>> >> > and halt the broker.
>> >> > Also, I've seen old TemporaryQueues that were removed minutes ago
>> come
>> >> > back and start queueing buffered messages when I start purging other
>> >> > consumerless TempQueues.
>> >> >
>> >> >
>> >> > In short I think there are two problems here.
>> >> > First, the detection of the client disconnection is not instant on
>> the
>> >> > server. This makes it possible to send messages to the TemporaryQueue
>> >> > without getting an exception after it should have been removed.
>> >> >
>> >> > The second is that the TemporaryQueue is able to resurrect after its
>> >> > creator/consumer has left. In other words it has outlived its
>> >> connection.
>> >> >
>> >> >
>> >> > I attach a simple client and server that are able to reproduce this
>> >> > reliably (at least on my computer).
>> >> > Just run the server and maybe two clients. Restart the clients
>> perhaps
>> >> 1-2
>> >> > times per minute and the issue will show pretty quickly. At least on
>> my
>> >> > P4, maybe a computer with multiple cores will behave different.
>> >> >
>> >> > Regards
>> >> > Zemus
>> >> >
>> >> >  http://old.nabble.com/file/p27130529/TestClient.javaTestClient.java
>> >> >  http://old.nabble.com/file/p27130529/ClientShutdownHook.java
>> >> > ClientShutdownHook.java
>> >> >
>> >> >  http://old.nabble.com/file/p27130529/TestServer.javaTestServer.java
>> >> >  http://old.nabble.com/file/p27130529/ServerShutdownHook.java
>> >> > ServerShutdownHook.java
>> >> >
>> >>
>> >>
>> >> After more investigation I've changed my mind about the nature of the
>> >> problem. The prefetch limit is not involved.
>> >>
>> >> My new theory goes like this:
>> >>
>> >> 1. The temporary queue is removed correctly when the client exists and
>> >> closes its connection.
>> >>
>> >> 2. The server might receive an exception when it tries to send to the
>> >> removed temporary queue. BUT sometimes it doesn't get that exception
>> >> (perhaps because the broker was soo busy that it didn't have time to
>> >> notify
>> >> the activemq part of the server in time) and instead the broker creates
>> a
>> >> new queue with the same name, although without any consumer, when the
>> >> server
>> >> sends the next message to the stored destination.
>> >>
>> >> 3. The server knows nothing about this and keeps sending messages that
>> >> pile
>> >> up in the broker, which in turn, eventually runs out of memory (at
>> least
>> >> jconsole reports MemoryPercentageUsed 100 for the broker object).
>> >>
>> >>
>> >> If this sounds reasonable, my questions are instead:
>> >>
>> >> * How can I prevent the broker from creating a new temporary queue
>> >> because
>> >> of the server's late message?
>> >>
>> >> * Would listening to advisory support for the temp queue at the server
>> >> side
>> >> notify me in time to stop the sending of any more messages when the
>> temp
>> >> queue is removed?
>> >>
>> >> * Or is there a better way that would stop the server from sending a
>> >> message
>> >> to a removed temp queue?
>> >>
>> >> Regards
>> >> Zemus
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27147013.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Problems-with-prefetch-and-TemporaryQueues-tp27130529p27162131.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

--00032555b682137ec3047d23c8e7--