activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eduardo Corral <ecor...@experienceis.com>
Subject Broker hangs under heavy load: possible TCP/IP problems?
Date Mon, 01 Sep 2008 13:17:32 GMT
Hi everyone

We are using ActiveMQ 4.1.1 over JRE 1.5.0_06 and Solaris 10 as JMS 
provider. Our broker is embedded inside the application, and we use it, 
not only to hold destinations for incoming external messages, but to 
hold destinations for internal communication (producers and consumers 
executing within the same JVM, in different threads).

Under heavy load, sometimes the broker hangs on a socketWrite0 call:

"Session(recv,TaskManagerQueue)#56" prio=10 tid=0x00b4a6d8 nid=0x6a 
runnable [0xeb9fe000..0xeb9ffaa8]
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(Unknown Source)
    at java.net.SocketOutputStream.write(Unknown Source)
    at 
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
    at java.io.DataOutputStream.flush(Unknown Source)
    at 
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
    at 
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:145)
    at 
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
    at 
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
    at 
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:392)
    - locked <0x2abab308> (a java.lang.Object)
    at 
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
    - locked <0x2aba90f0> (a java.lang.Object)
    at 
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
    at 
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1155)
    at 
org.apache.activemq.TransactionContext.begin(TransactionContext.java:201)
    at 
org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1564)
    at 
org.apache.activemq.ActiveMQMessageConsumer.ackLater(ActiveMQMessageConsumer.java:699)
    at 
org.apache.activemq.ActiveMQMessageConsumer.beforeMessageIsConsumed(ActiveMQMessageConsumer.java:651)
    at 
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:487)
    ...

MutexTransport is locking 0x2aba90f0, and thus making other threads to stop:

"Session(recv,TaskManagerQueue)#49" prio=10 tid=0x00b177f0 nid=0x63 
waiting for monitor entry [0xec0ff000..0xec0ffa28]
    at 
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:46)
    - waiting to lock <0x2aba90f0> (a java.lang.Object)
    at 
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
    at 
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1155)
    at 
org.apache.activemq.TransactionContext.begin(TransactionContext.java:201)
    at 
org.apache.activemq.ActiveMQSession.doStartTransaction(ActiveMQSession.java:1564)
    at 
org.apache.activemq.ActiveMQMessageConsumer.ackLater(ActiveMQMessageConsumer.java:699)
    at 
org.apache.activemq.ActiveMQMessageConsumer.beforeMessageIsConsumed(ActiveMQMessageConsumer.java:651)
    at 
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:487)
    ...

socketWrite0 would never stop, and, eventually, all consuming threads 
will block waiting for the same object. This thread is consuming from a 
local destination (within the embedded broker).

Don't know if it's related, but another thread is stuck in the same call 
(IP changed, but it's a non-localhost IP):

"ActiveMQ Transport: tcp:///XXX.XXX.XXX.XXX:32469" daemon prio=10 
tid=0x01a0be60 nid=0x25e23 runnable [0x1217e000..0x1217fa28]
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(Unknown Source)
    at java.net.SocketOutputStream.write(Unknown Source)
    at 
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:109)
    at java.io.DataOutputStream.flush(Unknown Source)
    at 
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:119)
    at 
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:145)
    at 
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
    at 
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
    at 
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
    - locked <0x2abbd550> (a java.lang.Object)
    at 
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1138)
    at 
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:805)
    at 
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
    at 
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:360)
    at 
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:83)
    at 
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:51)
    at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:417)
    at org.apache.activemq.broker.region.Topic.send(Topic.java:264)
    at 
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:292)
    at 
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:385)
    at 
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:193)
    at 
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:277)
    at 
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:242)
    at 
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:237)
    at 
org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:190)
    at 
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221)
    at 
org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:229)
    at 
org.apache.activemq.broker.TransportConnection.processRemoveDestination(TransportConnection.java:530)
    - locked <0x2e5d8a38> (a 
org.apache.activemq.broker.jmx.ManagedTransportConnection)
    at 
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:123)
    at 
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
    at 
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
    at 
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
    at 
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
    at 
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:124)
    at 
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
    at 
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
    at java.lang.Thread.run(Unknown Source)

Any ideas about this? Searching about this I found it could be a TCP/IP 
related problem: the socketWrite0 could be locked because the socket's 
send buffer is full, and the other end is no longer reading (whichever 
the reason: dead reader, slow reader). Should failover take care of this?

Any info would be highly appreciated.
Best regards
Eduardo Corral


Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message