Return-Path: Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: (qmail 54288 invoked from network); 17 Aug 2009 22:53:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Aug 2009 22:53:39 -0000 Received: (qmail 72687 invoked by uid 500); 17 Aug 2009 22:53:58 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 72618 invoked by uid 500); 17 Aug 2009 22:53:58 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 72608 invoked by uid 99); 17 Aug 2009 22:53:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 22:53:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 22:53:56 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id CE5BA234C044 for ; Mon, 17 Aug 2009 15:53:35 -0700 (PDT) Message-ID: <1399841019.1250549615835.JavaMail.jira@brutus> Date: Mon, 17 Aug 2009 15:53:35 -0700 (PDT) From: "Mats Henrikson (JIRA)" To: dev@activemq.apache.org Subject: [jira] Created: (AMQ-2356) optimizeDispatch=true and UseDedicatedTaskRunner=false lead to deadlocked queues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: ae95407df07c98740808b2ef9da0087c X-Virus-Checked: Checked by ClamAV on apache.org optimizeDispatch=true and UseDedicatedTaskRunner=false lead to deadlocked queues -------------------------------------------------------------------------------- Key: AMQ-2356 URL: https://issues.apache.org/activemq/browse/AMQ-2356 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.2.0 Reporter: Mats Henrikson We have an environment where we have a very large number of destinations. In an effort to reduce the number of threads I have set the options -Dorg.apache.activemq.UseDedicatedTaskRunner=false and Unfortunately this very quickly leads to deadlocked queues. My environment is: ActiveMQ 5.2 Ubunty Jaunty kernel 2.6.28-14-generic #47-Ubuntu SMP (although only a single core on my system) TCP transportConnector To reproduce the bug (which I can do 100% of the time) I connect 5 consumers (AUTO_ACK) to 5 different queues. Then I start 5 producers and pair them up with a consumer on a queue, and they start sending PERSISTENT messages. I've set the producer to send 100 messages and disconnect, and the consumer to receive 100 messages and disconnect. The first pair usually gets through their 100 messages and disconnect, at which point all the other pairs have deadlocked at less than 30 messages each. At this point I can connect jconsole and hit the Detect Deadlock button and it finds the 4 deadlocks for me. The 4 deadlocks all have the same stacktraces, one of which looks like this: {panel} Name: ActiveMQ Transport: tcp:///127.0.0.1:49307 State: BLOCKED on java.lang.Object@f8828 owned by: ActiveMQ Transport: tcp:///127.0.0.1:48031 Total blocked: 3 Total waited: 1 Stack trace: org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:576) org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150) org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266) org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1230) org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308) org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011) org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1146) org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1141) org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:474) org.apache.activemq.broker.region.Queue.send(Queue.java:417) org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350) org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437) org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224) org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126) org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95) org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126) ...SpecialAuthorizationBroker.send(DynamicAuthorizationBroker.java:156) org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126) org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133) org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450) org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639) org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305) org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143) org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206) org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84) org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203) org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185) java.lang.Thread.run(Thread.java:595) ---------------------------------------------------------------------------------- Name: ActiveMQ Transport: tcp:///127.0.0.1:48031 State: BLOCKED on org.apache.activemq.broker.region.Queue$2@102d82c owned by: ActiveMQ Transport: tcp:///127.0.0.1:49307 Total blocked: 3 Total waited: 0 Stack trace: org.apache.activemq.broker.region.Queue.iterate(Queue.java:951) org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1146) org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1073) org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:51) org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:238) org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373) org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462) org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194) org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85) org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456) org.apache.activemq.command.MessageAck.visit(MessageAck.java:205) org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305) org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143) org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206) org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84) org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203) org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185) java.lang.Thread.run(Thread.java:595) {panel} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.