activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Torsten Mielke (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5692) Inactivity monitor does not time out on stuck socket writes
Date Tue, 07 Apr 2015 11:58:12 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483067#comment-14483067
] 

Torsten Mielke commented on AMQ-5692:
-------------------------------------

Attached reproducer in AMQ-5692.pl. 
To reproduce issue follow these steps:

- Run broker with stomp connector on port 61613 and this connector configuration
{code:xml}
<transportConnector name="stomp"  uri="stomp://0.0.0.0:61613?transport.defaultHeartBeat=10000,0&amp;transport.useKeepAlive=true&amp;trace=true"/>
{code}
- Run consumer 
{code}
perl ./AMQ-5692.pl consumer
{code}

- Run producer 
{code}
perl ./AMQ-5692.pl producer
{code}

After consuming around 8000 messages, the consumer hangs and won't report any more messages
consumed. 
Take a broker thread dump and observe two stuck threads as in (1) below.
Despite the configured stomp heart-beat defaultHeartBeat=10000,0 the connection does not time
out after 10 seconds and no keep-alive is sent either. 

Have not managed to reproduce the issue with the ActiveMQ stomp client library. 

(1)
{code}
"ActiveMQ BrokerService[localhost] Task-4" daemon prio=5 tid=0x00007fe9ca087800 nid=0x650b
runnable [0x00000001192d9000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
	at org.apache.activemq.transport.stomp.StompTransportFilter.sendToStomp(StompTransportFilter.java:98)
	at org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:103)
	at org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:870)
	at org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:62)
	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:304)
	at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
	at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
	at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1419)
	at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:938)
	at org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:984)
	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

"ActiveMQ Transport: tcp:///192.168.178.28:60902@61613" daemon prio=5 tid=0x00007fe9ca086800
nid=0x3b0f waiting on condition [0x0000000118f2d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007c0ecc2f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
	at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:43)
	at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
	at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87)
	at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199)
	at org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:433)
	at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:247)
	at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	at java.lang.Thread.run(Thread.java:744)
{code}

>  Inactivity monitor does not time out on stuck socket writes
> ------------------------------------------------------------
>
>                 Key: AMQ-5692
>                 URL: https://issues.apache.org/jira/browse/AMQ-5692
>             Project: ActiveMQ
>          Issue Type: Improvement
>          Components: Broker
>    Affects Versions: 5.11.1
>            Reporter: Torsten Mielke
>              Labels: broker, inactivity
>         Attachments: AMQ-5692.pl
>
>
> It is possible that a socket write is stuck but the inactivity monitor currently does
not time out on a socketWrite. 
> {code:title=AbstractInactivityMonitor.java}
>     final void writeCheck() {
>         if (inSend.get()) {
>             LOG.trace("Send in progress. Skipping write check.");
>             return;
>         }
> {code}
> As a result a connection that is stuck in a tcp write will never be taken down due to
inactivity. If a client misbehaves the broker will not be able to clear that connection as
part of the inactivity monitoring.
> Now AMQ-2511 introduced a counter on the reachCheck() to detect it a socket read in progress
really retrieves data or is stuck. 
> I propose for a similar mechanism being applied on the writeCheck() operation so that
a socket write that is stuck can be detected and the connection can be closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message