activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: ActiveMQ Transport Threads Getting Blocked
Date Tue, 27 Aug 2013 03:33:37 GMT
Check that -- looks like the broker is trying to complete a sync send by
sending back a response and getting stuck on writing to the socket.


On Mon, Aug 26, 2013 at 8:02 PM, Christian Posta
<christian.posta@gmail.com>wrote:

> Yep, looks like you might have some slow topic consumers.
>
>
> On Mon, Aug 26, 2013 at 7:50 PM, davidk <davidklosowski@livenation.com>wrote:
>
>> I'm having issues with one of our activemq servers.  Many of the transport
>> threads are blocking after the server comes up and clients connect to it.
>>
>> The ActiveMQ version is 5.4.2 with quite a few back ported patches.
>>
>> There are many blocked transport threads and the snippet below just shows
>> the blocked thread and the running thread that is blocking it.  Judging by
>> the snippet from below, it looks like the Transport thread on
>> ssl:///10.73.216.103:51640 has the transport mutex and is not releasing
>> it.
>> It looks to be writing to a socket as a result of a dispatch but somehow
>> stuck.
>>
>> Does anyone have suggestions as to what may be causing this?
>>
>> "ActiveMQ Transport: ssl:///10.73.111.127:49523" - Thread t@7246
>>    java.lang.Thread.State: BLOCKED
>>         at
>>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>>         - waiting to lock <7fed4b35> (a java.lang.Object) owned by
>> "ActiveMQ
>> Transport: ssl:///10.73.216.103:51640" t@663
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1284)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:822)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:782)
>>         at
>>
>> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:525)
>>         at
>>
>> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:99)
>>         at
>>
>> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
>>         at
>> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:657)
>>         at
>> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:489)
>>         - locked <3bfdff07> (a org.apache.activemq.broker.region.Topic)
>>         at org.apache.activemq.broker.region.Topic.send(Topic.java:425)
>>         at
>>
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
>>         at
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:518)
>>         at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:491)
>>         at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:422)
>>         at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:417)
>>         at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:186)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> com.ticketmaster.activemq.security.IdentityBrokerFilter.addDestinationInfo(IdentityBrokerFilter.java:473)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:221)
>>         at
>>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:227)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:485)
>>         at
>>
>> org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:312)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:186)
>>         at
>>
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>         at
>>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>>         at
>>
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)
>>         - locked <5f718b0b> (a
>> org.apache.activemq.transport.InactivityMonitor$1)
>>         at
>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>>         at
>>
>> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>>         at
>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
>>         at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>    Locked ownable synchronizers:
>>         - None
>>
>>
>> "ActiveMQ Transport: ssl:///10.73.216.103:51640" - Thread t@663
>>    java.lang.Thread.State: RUNNABLE
>>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>>         at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>         at
>>
>> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:358)
>>         at
>> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:346)
>>         at
>>
>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:781)
>>         at
>>
>> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:753)
>>         at
>>
>> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
>>         - locked <48a49bda> (a
>> com.sun.net.ssl.internal.ssl.AppOutputStream)
>>         at
>>
>> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
>>         at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>>         at
>>
>> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:182)
>>         at
>>
>> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:255)
>>         - locked <2ef42a78> (a java.util.concurrent.atomic.AtomicBoolean)
>>         at
>>
>> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
>>         at
>>
>> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
>>         at
>>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>>         - locked <7fed4b35> (a java.lang.Object)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1284)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:822)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:782)
>>         at
>>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
>>         at
>>
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>         at
>>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>>         at
>>
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)
>>         - locked <2010445d> (a
>> org.apache.activemq.transport.InactivityMonitor$1)
>>         at
>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
>>         at
>>
>> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91)
>>         at
>>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
>>         at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>    Locked ownable synchronizers:
>>         - locked <bf8461d> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Transport-Threads-Getting-Blocked-tp4670703.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message