activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Tully (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AMQ-1976) Network of brokers gets out of sync for dispatched messages causing a shutdown of the bridge connections and stalling the brokers
Date Wed, 15 Oct 2008 16:25:52 GMT

    [ https://issues.apache.org/activemq/browse/AMQ-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46480#action_46480
] 

Gary Tully commented on AMQ-1976:
---------------------------------

are the producers using asyncSend?

I don't know the relevance of ERROR RecoveryListenerAdapter ... but I think the assertAckMatchesDispatched
is being a little harsh. My problem is reproducing the behavior.
messages that passe through a network are acked one by one. The error is telling us that the
ack is ocurring out of order; since the message was dispatched, 730 other messages have been
dispatched and it looks like the ack is for the 730th message. It is a valid ack though.
I think this can only happen in asyncSend, otherwise, each dispatched message is synchronously
acked through to its next hop.

Ideally, it would be great if you could produce a test case. I have been trying variants of
the NetworkLoadTest but am not reproducing the problem.
What does your producer look like, from the text above, it seems like there is a bunch of
activity on each broker?








> Network of brokers gets out of sync for dispatched messages causing a shutdown of the
bridge connections and stalling the brokers
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-1976
>                 URL: https://issues.apache.org/activemq/browse/AMQ-1976
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.2.0
>            Reporter: Jayson Minard
>
> Running 5.2.0 RC2 + patch AMQ-1971 
> I have a network of 5 brokers, running around 50,000 messages per minute per broker within
each broker and sending across a total of around 20,000 messages per minute to one of the
5 brokers via a bridge connection.  The configuration is the 5.2.0 default with broker names
changed on each config and with a network connector set for autodiscovery.  
> The system runs for a while then eventually on one broker (the receiving broker that
is the only one with the consumers for this queue) these errors appear:
> ERROR RecoveryListenerAdapter        - Message id ID:ingest1.mag.ss.com-54179-1223898804441-0:6:1:1:6134
could not be recovered from the data store - already dispatched
> ERROR RecoveryListenerAdapter        - Message id ID:query1.mag.ss.com-38822-1223898893819-0:58:16388:1:48
could not be recovered from the data store - already dispatched
> They appear in really high counts.
> On the sending side brokers (publishers but no consumers on these brokers for this queue),
these errors appear:
> INFO  DemandForwardingBridge         - query2.mag.ss.com bridge to query1.mag.ss.com
stopped
> INFO  DiscoveryNetworkConnector      - Establishing network connection between from vm://query2.mag.ss.com
to tcp://query1.mag.ss.com:61616
> INFO  DemandForwardingBridge         - Network connection between vm://query2.mag.ss.com#30
and tcp://query1.mag.ss.com/10.100.0.101:61616(query1.mag.ss.com) has been established.
> ERROR Service                        - Async error occurred: javax.jms.JMSException:
Unmatched acknowledege: Expected message count (1) differs from count in dispatched-list (732)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1) differs from
count in dispatched-list (732)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> 	at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> 	at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> 	at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> 	at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> 	at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> 	at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
> 	at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
> 	at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> 	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
> 	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> 	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:113)
> 	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> 	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> 	at org.apache.activemq.network.DemandForwardingBridgeSupport$8.onCompletion(DemandForwardingBridgeSupport.java:643)
> 	at org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:61)
> 	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
> 	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> 	at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
> 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184)
> 	at java.lang.Thread.run(Thread.java:619)
> INFO  DemandForwardingBridge         - Network connection between vm://query2.mag.ss.com#30
and tcp://query1.mag.ss.com/10.100.0.101:61616 shutdown due to a local error: javax.jms.JMSException:
Unmatched acknowledege: Expected message count (1) differs from count in dispatched-list (732)
> INFO  DemandForwardingBridge         - query2.mag.ss.com bridge to query1.mag.ss.com
stopped
> INFO  DiscoveryNetworkConnector      - Establishing network connection between from vm://query2.mag.ss.com
to tcp://query1.mag.ss.com:61616
> INFO  DemandForwardingBridge         - Network connection between vm://query2.mag.ss.com#32
and tcp://query1.mag.ss.com/10.100.0.101:61616(query1.mag.ss.com) has been established.
> ERROR Service                        - Async error occurred: javax.jms.JMSException:
Unmatched acknowledege: Expected message count (1) differs from count in dispatched-list (730)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1) differs from
count in dispatched-list (730)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> 	at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> 	at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> 	at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> 	at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> 	at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
> 	at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
> 	at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
> 	at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
> 	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> 	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> 	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
> 	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> 	at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:113)
> 	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> 	at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> 	at org.apache.activemq.network.DemandForwardingBridgeSupport$8.onCompletion(DemandForwardingBridgeSupport.java:643)
> 	at org.apache.activemq.transport.FutureResponse.set(FutureResponse.java:61)
> 	at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
> 	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> 	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> 	at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> 	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
> 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:184)
> 	at java.lang.Thread.run(Thread.java:619)
> INFO  DemandForwardingBridge         - Network connection between vm://query2.mag.ss.com#32
and tcp://query1.mag.ss.com/10.100.0.101:61616 shutdown due to a local error: javax.jms.JMSException:
Unmatched acknowledege: Expected message count (1) differs from count in dispatched-list (730)
> ERROR Service                        - Async error occurred: javax.jms.JMSException:
Unmatched acknowledege: Expected message count (1) differs from count in dispatched-list (729)
> javax.jms.JMSException: Unmatched acknowledege: Expected message count (1) differs from
count in dispatched-list (729)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:445)
> 	at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:187)
> 	at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
> 	at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
> 	at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
> ...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message