activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakub (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-6666) Failover Transport - send timeout not working
Date Fri, 21 Jul 2017 10:17:00 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096095#comment-16096095
] 

Jakub edited comment on AMQ-6666 at 7/21/17 10:16 AM:
------------------------------------------------------

In my case it also occurs in 5.14.5. I use below configuration:
* JDBC persistence storage 
* master - slave 
* client uses failover protocol
* AMQ is part of XA transaction (2PC)

It leads to hang of entire application, because one of thread owns forever a lock:
{{
java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
        - locked <0x000000073ec21360> (a java.lang.Object)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
        at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
        at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1428)
        at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:751)
        at org.apache.activemq.TransactionContext.invokeBeforeEnd(TransactionContext.java:424)
        at org.apache.activemq.TransactionContext.end(TransactionContext.java:408)
        at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:61)
        at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:588)
        at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:567)
        at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:414)
        at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:262)
        at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
        at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1020)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
        at org.apache.aries.transaction.internal.AriesPlatformTransactionManager.commit(AriesPlatformTransactionManager.java:75)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:245)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1164)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1156)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1053)
        at java.lang.Thread.run(Thread.java:748)
}}


was (Author: ragnor84):
In my case it also occurs in 5.14.5. I use below configuration:
* JDBC persistence storage 
* master - slave 
* client uses failover protocol
* AMQ is part of XA transaction (2PC)

> Failover Transport - send timeout not working
> ---------------------------------------------
>
>                 Key: AMQ-6666
>                 URL: https://issues.apache.org/jira/browse/AMQ-6666
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.14.0
>            Reporter: Martin Lichtin
>            Priority: Critical
>
> Running into a situation with the Failover Transport not respecting the timeout that's
been set. The symptom is endless messages of this kind:
> {noformat}
> 2017-04-29 09:48:26,128 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,228 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,329 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,429 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:26,530 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> ...
> 2017-04-29 09:48:33,270 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> 2017-04-29 09:48:33,371 | TRACE | .engine.cfgengine.in]-11 | FailoverTransport      
         | sport.failover.FailoverTransport 615 | 81 - org.apache.activemq.activemq-osgi -
5.14.0 | Waiting for transport to reconnect..: TransactionInfo {commandId = 127798, responseRequired
= true, type = 7, connectionId = ID:inucdev4-57330-1493370444659-3:3, transactionId = XID:[1096044365,globalId=6374726c6366672d656e67696e653130333530323030303034,branchId=6374726c6366672d656e67696e6531313036383134]}
> {noformat}
> The code seems to never get out of this loop:
> {noformat}
>     while (transport == null && !disposed && connectionFailure == null
>             && !Thread.currentThread().isInterrupted() && willReconnect())
{
>         LOG.trace("Waiting for transport to reconnect..: {}", command);
>         long end = System.currentTimeMillis();
>         if (command.isMessage() && timeout > 0 && (end - start >
timeout)) {
>             timedout = true;
>             LOG.info("Failover timed out after {} ms", (end - start));
>             break;
>         }
>         try {
>             reconnectMutex.wait(100);
>         } catch (InterruptedException e) {
>             Thread.currentThread().interrupt();
>             LOG.debug("Interupted:", e);
>         }
>         transport = connectedTransport.get();
>     }
> {noformat}
> The timeout is set to 5000ms and should have hit a long time ago, but as "command.isMessage()"
returns false, it endlessly stays inside the loop.
> The "command.isMessage()" condition should likely be removed.
> Currently running tests with a patched ActiveMQ, and the situation has improved and fail-over
seems to have worked (mostly). (Only seeing an issue with a topic consumer that has not reconnected.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message