activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Vicary <nations...@btclick.com>
Subject ActiveMQ and Durable Topic subscriptions after subscriber is uncleanly terminated
Date Wed, 18 Jul 2007 09:14:23 GMT

Hi all,

I have been having some issues which are currently show-stoppers with the
use of Durable subscriptions with Active MQ Topics for our large scale
integration project.

I've been writing the subscriber in C#, but the issue also remains for the
Java implementation. The standard approach for establishing a durable
subscriber is to perform all the standard steps for setting up a subscriber,
along with setting a ClientID (to provide the unique ID for the subscriber
application), and calling CreateDurableConsumer on the session.

On the first attempt, the subscription is established, and the messages are
correctly received.  If the subscriber is then shutdown in a controlled
manner manor, and the connection is correctly stopped and closed, then
subsequent restarts of the subscriber will perform as expected. Great, all
works fine.

However, under real failure scenarios (machine goes pop, or goes offline for
some reason resulting in a subscriber restart) the connection doesn't have
chance to correctly terminate the connection with the broker - essentially
an "uncontrolled shutdown".  This is where the problem arises.  If the
subscriber now attempts to establish a durable subscription with the same
ClientID and name as before, the broker returns a 'Client XXX already
connected' error, and prevents the connection from being made - even though
the previous client/subscriber is not actually connected, or even running. 
This doesn’t seem to be time bound either - even waiting for a period of
time (minutes) and retrying, will produce the same results, so it's not the
socket in TIME_WAIT state which is causing it.

After further investigation, I've discovered the following:

Using Jconsole to look into the state of the broker, it seems that,
following an uncontrolled client disconnect (as previously performed) the
previously created Connection instance is still classed (by the broker) as
being both live and connected, although it blatantly isn't connected (or
even live), because the client is no longer there.  This is persisted by the
broker, and never seems to be cleared (until a broker restart, which is
unacceptable in an enterprise scale environment just to recover from a
single subscriber failure)

The broker should detect the socket disconnect from the failed subscriber,
and clean up the connection status in the broker.

Another observation is that if the connection is manually cleared using
Jconsole (using the relevant operation on the connection instance), the
subscriber can indeed reconnect using the durable subscription.

Another observation is that this only happens if NO messages are published
to the topic during the subscriber downtime.  If however a message is
published to the topic during the subscriber downtime, the broker will
detect that the subscriber is no longer live, and clear up the connection. 
This results in the subscriber being able to reconnect successfully. 
However, in production environments, we cannot guarantee that a message will
be sent on a topic during the subscriber downtime - although most topics
will have high utilisation, some have low throughput - but this cannot be
relied upon, and the failure of a single durable subscription will result in
the failure of the complete subscribing application.

It seems that all the ActiveMQ unit tests (or the ones I've looked at) to
test the durability of the connection, perform orderly shutdown of the
connection during the test.  This results in the broker correctly cleaning
the connection status, and the remaining tests being successful.

Under other JMS implementations (namely Tibco EMS but I've performed similar
in the past with JBossMQ), this doesn't happen.  Many JMS resources specify
that if a durable subscription is attempted and one is already established,
then the existing subscription is overwritten, and the new one is
established.  This doesn't seem to be the case with ActiveMQ - instead it
throws an exception.

My main questions to the ActiveMQ forum are:
1) Is there a workaround for this to allow subsequent durable subscriptions
to work following an "uncontrolled" subscriber shutdown?
2) Does ActiveMQ have a configuration parameter to allow subsequent durable
subscriptions to overwrite existing ones (even if the existing ones are
actually dead connections)
3) Is there anything within ActiveMQ which can periodically test the
connections in the broker to see if they are still live - if not, then clean
them up to overcome this problem
4) Has anybody else experienced this issue in a production quality
environment or otherwise - I've seen many posts to do with 'Client XXX
already connected' but nothing which resolves the issue other than 'fixed in
the 4.1…. Release'.  We are using 4.1.1 so we should see the fix - this
sounds like another issue which has slipped though the net.

Any feedback on this would be much appreciated.

Kind regards

Simon Vicary
Integration and Technical Delivery Lead.
-- 
View this message in context: http://www.nabble.com/ActiveMQ-and-Durable-Topic-subscriptions-after-subscriber-is-uncleanly-terminated-tf4102045s2354.html#a11665143
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message