Return-Path: X-Original-To: apmail-activemq-dev-archive@www.apache.org Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7A30B174E5 for ; Tue, 7 Apr 2015 11:58:12 +0000 (UTC) Received: (qmail 15225 invoked by uid 500); 7 Apr 2015 11:58:12 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 15159 invoked by uid 500); 7 Apr 2015 11:58:12 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 15148 invoked by uid 99); 7 Apr 2015 11:58:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2015 11:58:12 +0000 Date: Tue, 7 Apr 2015 11:58:12 +0000 (UTC) From: "Torsten Mielke (JIRA)" To: dev@activemq.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AMQ-5692) Inactivity monitor does not time out on stuck socket writes MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AMQ-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483067#comment-14483067 ] Torsten Mielke commented on AMQ-5692: ------------------------------------- Attached reproducer in AMQ-5692.pl. To reproduce issue follow these steps: - Run broker with stomp connector on port 61613 and this connector configuration {code:xml} {code} - Run consumer {code} perl ./AMQ-5692.pl consumer {code} - Run producer {code} perl ./AMQ-5692.pl producer {code} After consuming around 8000 messages, the consumer hangs and won't report any more messages consumed. Take a broker thread dump and observe two stuck threads as in (1) below. Despite the configured stomp heart-beat defaultHeartBeat=10000,0 the connection does not time out after 10 seconds and no keep-alive is sent either. Have not managed to reproduce the issue with the ActiveMQ stomp client library. (1) {code} "ActiveMQ BrokerService[localhost] Task-4" daemon prio=5 tid=0x00007fe9ca087800 nid=0x650b runnable [0x00000001192d9000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176) at org.apache.activemq.transport.stomp.StompTransportFilter.sendToStomp(StompTransportFilter.java:98) at org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:103) at org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:870) at org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:62) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:304) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286) at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1419) at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:938) at org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:984) at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) "ActiveMQ Transport: tcp:///192.168.178.28:60902@61613" daemon prio=5 tid=0x00007fe9ca086800 nid=0x3b0f waiting on condition [0x0000000118f2d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007c0ecc2f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:43) at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270) at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:87) at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:199) at org.apache.activemq.transport.stomp.ProtocolConverter.onStompAck(ProtocolConverter.java:433) at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:247) at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:75) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) at java.lang.Thread.run(Thread.java:744) {code} > Inactivity monitor does not time out on stuck socket writes > ------------------------------------------------------------ > > Key: AMQ-5692 > URL: https://issues.apache.org/jira/browse/AMQ-5692 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker > Affects Versions: 5.11.1 > Reporter: Torsten Mielke > Labels: broker, inactivity > Attachments: AMQ-5692.pl > > > It is possible that a socket write is stuck but the inactivity monitor currently does not time out on a socketWrite. > {code:title=AbstractInactivityMonitor.java} > final void writeCheck() { > if (inSend.get()) { > LOG.trace("Send in progress. Skipping write check."); > return; > } > {code} > As a result a connection that is stuck in a tcp write will never be taken down due to inactivity. If a client misbehaves the broker will not be able to clear that connection as part of the inactivity monitoring. > Now AMQ-2511 introduced a counter on the reachCheck() to detect it a socket read in progress really retrieves data or is stuck. > I propose for a similar mechanism being applied on the writeCheck() operation so that a socket write that is stuck can be detected and the connection can be closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)