activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anuj Khandelwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5357) Race conditions in ActiveMQ Connection Pooling
Date Wed, 24 Sep 2014 06:57:34 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145991#comment-14145991
] 

Anuj Khandelwal commented on AMQ-5357:
--------------------------------------

Hi Tim,

I have tried to reproduce. It's not the exact test case but kind of similar problem. 

1. I have my own local plugin "ActiveMQLoggingPlugin extends BrokerPluginSupport" and override
"removeConnections" and "removeSession" functions.
2. I returned from the functions without passing it to super class for example: 
-----------
    public void removeConnection(ConnectionContext context, ConnectionInfo info, Throwable
error) throws Exception
    {
        if (info.toString() != "hello") {
            return; // It's a hack so that it always return from here. 
        }
        super.removeConnection(context, info, error);
    }

3. Now when I run the client application(consumers) which is using pooledConnectionFactory
with maxConnection=5. it is creating 5 different connections using different client ids. Each
listeners is a durable subscriber and I am setting different client id for each of them (It
is using the configuration mentioned in previous comment)

4. It is able to connect and work properly. Now I stop the client application. I can see that
connections are gone and applciation is stopped. I checked that broker is not connecting to
any client now. 

5 Now when I start the client application. I get the invalidclientIDexception by saying that
this client is already connected to the broker. 
I think since broker is not closing the connection, it keeps the client id associated with
it and does not clear the connection state. and when next time application starts, it says
that client is already connected even if no client is actually connected to the broker. 

Thanks,
Anuj 



> Race conditions in ActiveMQ Connection Pooling
> ----------------------------------------------
>
>                 Key: AMQ-5357
>                 URL: https://issues.apache.org/jira/browse/AMQ-5357
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: activemq-pool, Broker
>    Affects Versions: 5.10.0
>            Reporter: Anuj Khandelwal
>
> Hi,
> Broker version : 5.10.0
> Problem : 
> I am using ActiveMQv5.10.0 broker in my setup. In my broker I am facing a weird problem
where a durable subscriber is not able to connect to the broker because of stale subscription
shown by the broker with the same client id. Please note that this issue is not reproducible,
I tried but not able to reproduce. It happens sometimes. 
> Details: 
> In my application I am creating 3 durable subscriptions on different topics. I am using
PooledConnectionFactory with maxConnection=3 to create connections for each listener. I am
using separate Spring Default message listener container for creation of the each listener.

> Sometime when I start my client, I get below exception :  
> -----------
> listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful():907 ERROR]:
Could not refresh JMS Connection for destination 'topic://test.newPnlSpn' - retrying in 5000
ms. Cause: Broker: mqbrokerdev - Client: new_pnl_spn_test_client already connected from tcp://a.b.c.d:2338

> -----------
> But when I check connections from "a.b.c.d". I don;t see any connections from the IP.
Even on JConsole, it does not show any clients connecting to the same topic with this client_id.

> In some cases i observed that this is the IP address where my client was previously running.

> Because of this issue my current client which is running from some other IP address is
not able to connect to the broker. 
> I suspect that there is some race condition in ActiveMQ connection pooling. This is a
serious issue I am facing in my setup, My clients are not able to connect to the broker.
> When I face this issue :
> I tried to restart the broker by clearing out the persistent store, but this issue was
still there. It only got resolved when I changed the client ID of my listener. 
> Which means broker somewhere keeping stale connection state for this client id. As soon
as new client id comes, it start working properly and connecting to the broker.
> Detailed Error logs: 
> [20140915 16:23:14:964 IST (jmsContainer-6) org.springframework.jms.listener.DefaultMessageListenerContainer#handleListenerSetupFailure
666 ERROR] - Setup
>  of JMS message listener invoker failed - trying to recover 
> javax.jms.IllegalStateException: javax.jms.InvalidClientIDException: Broker: mqbrokerdev
- Client: new_pnl_spn_seed_client already connected from tcp://10
> *.*.*.*:3456
>         at org.apache.activemq.jms.pool.ConnectionPool.createSession(ConnectionPool.java:135)
>         at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
>         at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:200)
>         at org.springframework.jms.listener.DefaultMessageListenerContainer.access$1000(DefaultMessageListenerContainer.java:116)
>         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.initResourcesIfNecessary(DefaultMessageListenerCon
> tainer.java:883)
>         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.jav
> a:869)
>         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:810)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.jms.InvalidClientIDException: Broker: mqbrokerdev - Client: new_pnl_spn_seed_client
already connected from tcp://10.79.26.84:2338
>         at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:246)
>         at org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnection(ManagedRegionBroker.java:231)
>         at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
>         at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
>         at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:92)
>         at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
>         at org.apache.activemq.broker.util.LoggingBrokerPlugin.addConnection(LoggingBrokerPlugin.java:189)
>         at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
>         at tools.jms.ActiveMQLoggingPlugin.addConnection(ActiveMQLoggingPlugin.java:425)
>         at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:97)
>         at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:764)
>         at org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:79)
>         at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:139)
>         at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
>         at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)
>         at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
>         at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>         at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
>         at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>         at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
>         at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
>         ... 1 more
> Thanks,
> Anuj



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message