Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (nike.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <14200217.post@talk.nabble.com>
Date: Thu, 6 Dec 2007 12:21:03 -0800 (PST)
From: Jason Rosenberg <jrosenberg@shopzilla.com>
To: users@activemq.apache.org
Subject: Test-case for problem with connections not refreshing when pooled
 with failover
In-Reply-To: <ec6e67fd0711270415s3c5b7626re4d000052da02251@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <13292887.post@talk.nabble.com> <47189CDC.9050702@tid.es>
 <13293655.post@talk.nabble.com> <13293740.post@talk.nabble.com>
 <4718ABD1.3030702@tid.es> <13294714.post@talk.nabble.com>
 <13958172.post@talk.nabble.com>
 <ec6e67fd0711262309u72baca61y4a16fa00299a1e6f@mail.gmail.com>
 <13969349.post@talk.nabble.com>
 <ec6e67fd0711270415s3c5b7626re4d000052da02251@mail.gmail.com>


James,

I've created a test-case here, which demonstrates clearly the main problem,
with connections created with a pooledConnection don't recover quickly, when
a broker becomes available, when the failover transport is used, with a
maxReconnectAttempts setting.

Attached is a source tar file, which you can extract and build, with maven. 
It should build a web-app war file, which you can drop into tomcat (or
probably any j2ee container).  I have been using java 1.6 and maven 2.0.7,
although I don't know that those should be required.

The web-app implements a simple web-service (implemented with CXF).....

It exposes three methods:

/pingTcp
/pingFailover
/pintFailoverWithMaxAttempts

Which use respectively, these 3 broker uri's, respectively:

tcp://localhost:61616
failover:(tcp://localhost:61616,tcp://localhost:61626)?initialReconnectDelay=20&amp;maxReconnectDelay=500&amp;randomize=false
failover:(tcp://localhost:61616,tcp://localhost:61626)?initialReconnectDelay=20&amp;maxReconnectDelay=500&amp;randomize=false&amp;maxReconnectAttempts=2

All the ping method does is create a connection (without actually trying to
do anything with it).

Assuming you have the web-app deployed in tomcat, running on port 8080,
you'd hit the different methods as such:

http://localhost:8080/failover-test-web-1.0/FailoverTest/pingTcp
http://localhost:8080/failover-test-web-1.0/FailoverTest/pingFailover
http://localhost:8080/failover-test-web-1.0/FailoverTest/pingFailoverWithMaxAttempts


The connections are created using the activeMQ PooledConnectionFactory class
(see the spring template failover-test-ws.xml in the test-case source code). 
I originally was seeing the problem with the Jencks connection pooling, but
it appears to happen also using the activeMQ connection pooling too....

To reproduce the problem, do the following:

1. start with both activemq and tomcat down.
2. start tomcat with the webapp.
3. hit pingFailoverWithMaxAttempts link.
    it will return a connection error response.
4. Bring up the broker on localhost:61616
5. hit pingFailoverWithMaxAttempts link again, several times.
    it will continue returning the failed connection link.
6. Wait 3 minutes
7. hit pingFailoverWithMaxAttempts link again.
    it will finally return successfully, with a good connection

So, it takes 3 minutes for it to recognize a recovered broker.

Next, repeat the above, this time using the pingTcp link each time.  Notice
that it recovers immediately.

Also, when using the pingFailover method, it generally recovers immediately,
but has the downside that it hangs the response until the broker is
available (thus the desirability of the maxReconnectAttempts option).  There
has been suggestion in the forums to use a TransportListener, but it's not
obvious to me how one would do this, in a use-case similar to this example,
which uses a PooledConnectionFactory, etc.  (ideas would be welcome
there)....

Thanks for any help,

Jason


http://www.nabble.com/file/p14200217/failover_test_ws.source.tar
failover_test_ws.source.tar 


James.Strachan wrote:
> 
> On 27/11/2007, Jason Rosenberg <jrosenberg@shopzilla.com> wrote:
>> James,
>>
>> In this case, I am creating a connection for the purspose of sending a
>> message (so not a message listener).  I am using the Spring JmsOperations
>> template, and just injecting the connectionFactory property  for the
>> jmsOperations object on initialization.
>>
>> The connectionFactory is org.jencks.amqpool.PooledConnectionFactory.  I
>> presume it's up to the factory code to properly dispose of the failed
>> connection's resources.  I also had the same result configuring it with
>> the
>> ActiveMQ factory, org.apache.activemq.pool.PooledConnectionFactory.
>>
>> In each case the pooled connection factories in turn use a
>> 'connectionFactory' object of
>> org.apache.activemq.ActiveMQConnectionFactory....
>>
>> What I've noticed is that this configuration works fine, if I use a
>> simple
>> TcpTransport.  That is, if the connection fails, the connection factory
>> seems to dispose of the resources and create a fresh one for the next
>> attempt, and so it recovers quickly when the broker recovers.
>>
>> Somehow, the FailoverTransport must fail in a different way, since the
>> connection factory doesn't detect that it needs to remove it and create a
>> new object on the next request.
>>
>> Finally, I've also seen that the FailoverTransport, if configured without
>> maxReconnectAttempts, doesn't shutdown cleanly, if there's no broker
>> present.  It hangs, or throws an exception.   I provide the
>> destroy-method="stop" on the pooled connection factory object.
>>
>> Thanks for any help,
> 
> Do you have a test case that demonstrates your issue?
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Timeout-and-Failover-on-a-queue-tf4652630s2354.html#a14200217
Sent from the ActiveMQ - User mailing list archive at Nabble.com.