Return-Path: Delivered-To: apmail-activemq-dev-archive@www.apache.org Received: (qmail 5856 invoked from network); 26 May 2010 12:04:19 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 May 2010 12:04:19 -0000 Received: (qmail 59754 invoked by uid 500); 26 May 2010 12:04:19 -0000 Delivered-To: apmail-activemq-dev-archive@activemq.apache.org Received: (qmail 59568 invoked by uid 500); 26 May 2010 12:04:18 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 59560 invoked by uid 99); 26 May 2010 12:04:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 12:04:17 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 12:04:14 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o4QC3qxE008016 for ; Wed, 26 May 2010 12:03:53 GMT Message-ID: <29129800.3541274875432755.JavaMail.jira@thor> Date: Wed, 26 May 2010 08:03:52 -0400 (EDT) From: "Dennis Klinkott (JIRA)" To: dev@activemq.apache.org Subject: [jira] Commented: (AMQ-2749) Broker hangs in TopicSubscription.add In-Reply-To: <28882136.3081274809851786.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: ae95407df07c98740808b2ef9da0087c X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/activemq/browse/AMQ-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=59544#action_59544 ] Dennis Klinkott commented on AMQ-2749: -------------------------------------- I've attached our activemq.xml and a screenshot of a jmx console to a similar broker that is running since 5 days. we do not have any warnings or errors in the log. if we reached the temp store or memory limit on the pending message cursor, would we get a warning log entry? > Broker hangs in TopicSubscription.add > ------------------------------------- > > Key: AMQ-2749 > URL: https://issues.apache.org/activemq/browse/AMQ-2749 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.3.1 > Environment: red had enterprise > Reporter: Dennis Klinkott > Attachments: activemq.xml, jmx console screenshot.jpg, jstack.2010-05-25-14-01-30, jstack.2010-05-25-14-10-54 > > > Our broker suddenly hang in production environment. We created a bunch of jstacks to see what the reason is. > The jstacks show that the broker waits with many threads in doMessageSend: > e.g. > "ActiveMQ Transport: tcp:///10.231.233.117:51088" daemon prio=10 tid=0x00002aab84016800 nid=0x1f19 waiting for monitor entry [0x000000006a8f2000] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:402) > - waiting to lock <0x00002aaab46f8938> (a org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:384) > at org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:93) > at org.apache.activemq.broker.region.virtual.VirtualTopicInterceptor.send(VirtualTopicInterceptor.java:46) > at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445) > at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:443) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:373) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:368) > at org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:173) > at org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:214) > at org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:214) > at org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:221) > at org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:467) > at org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122) > at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300) > at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178) > at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) > at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:82) > at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:135) > at org.apache.activemq.transport.stomp.ProtocolConverter.createTempQueue(ProtocolConverter.java:611) > at org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertDestination(LegacyFrameTranslator.java:130) > at org.apache.activemq.transport.stomp.FrameTranslator$Helper.copyStandardHeadersFromFrameToMessage(FrameTranslator.java:93) > at org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertFrame(LegacyFrameTranslator.java:54) > at org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:591) > at org.apache.activemq.transport.stomp.ProtocolConverter.onStompSend(ProtocolConverter.java:231) > at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:173) > at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:71) > at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84) > at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204) > at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186) > at java.lang.Thread.run(Thread.java:619) > It is waiting to lock 0x00002aaab46f8938. This is the thread that holds the lock: > "ActiveMQ Transport: tcp:///10.231.233.117:43020" daemon prio=10 tid=0x00000000608d8800 nid=0xc6e in Object.wait() [0x000000007d41c000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:106) > - locked <0x00002aaab57d34e8> (a java.lang.Object) > at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) > - locked <0x00002aaab46f98e0> (a java.util.concurrent.CopyOnWriteArrayList) > at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:585) > at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:443) > - locked <0x00002aaab46f8938> (a org.apache.activemq.broker.region.Topic) > at org.apache.activemq.broker.region.Topic.send(Topic.java:384) > at org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:93) > at org.apache.activemq.broker.region.virtual.VirtualTopicInterceptor.send(VirtualTopicInterceptor.java:46) > at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354) > at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:445) > at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:443) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:373) > at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:368) > at org.apache.activemq.advisory.AdvisoryBroker.removeDestination(AdvisoryBroker.java:185) > at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:146) > at org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilter.java:146) > at org.apache.activemq.broker.MutableBrokerFilter.removeDestination(MutableBrokerFilter.java:153) > at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:713) > - locked <0x00002aaac32b10c0> (a org.apache.activemq.broker.jmx.ManagedTransportConnection) > at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:72) > at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:300) > at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178) > at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) > at org.apache.activemq.transport.stomp.StompTransportFilter.sendToActiveMQ(StompTransportFilter.java:82) > at org.apache.activemq.transport.stomp.ProtocolConverter.sendToActiveMQ(ProtocolConverter.java:135) > at org.apache.activemq.transport.stomp.ProtocolConverter.onStompDisconnect(ProtocolConverter.java:545) > at org.apache.activemq.transport.stomp.ProtocolConverter.onStompCommand(ProtocolConverter.java:189) > at org.apache.activemq.transport.stomp.StompTransportFilter.onCommand(StompTransportFilter.java:71) > at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84) > at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:204) > at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186) > at java.lang.Thread.run(Thread.java:619) > So it is the client 10.231.233.117 that has a stomp-connection to the broker, causes a TopicSubscription.add and causes the broker freeze. > This is how we configure our openwire transport-connector in the activemq.xml: > > > > > We took jstack-dumps with a distance of 9 minutes. Both are attached. In both dumps it is the same TopicSubscription.add that causes the hang. > How can we solve this issue? Looks like the connection should run into an timeout but doesn't. Is this an activemq bug? Can we somehow set the broker openwire timeout parameters? > Thanks for any hints. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.