From activemq-users-return-1626-apmail-geronimo-activemq-users-archive=geronimo.apache.org@geronimo.apache.org Wed May 31 11:17:33 2006 Return-Path: Delivered-To: apmail-geronimo-activemq-users-archive@www.apache.org Received: (qmail 71744 invoked from network); 31 May 2006 11:17:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 31 May 2006 11:17:33 -0000 Received: (qmail 72058 invoked by uid 500); 31 May 2006 11:17:32 -0000 Delivered-To: apmail-geronimo-activemq-users-archive@geronimo.apache.org Received: (qmail 72035 invoked by uid 500); 31 May 2006 11:17:32 -0000 Mailing-List: contact activemq-users-help@geronimo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: activemq-users@geronimo.apache.org Delivered-To: mailing list activemq-users@geronimo.apache.org Received: (qmail 72024 invoked by uid 99); 31 May 2006 11:17:32 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 May 2006 04:17:32 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of lists@nabble.com designates 72.21.53.35 as permitted sender) Received: from [72.21.53.35] (HELO talk.nabble.com) (72.21.53.35) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 May 2006 04:17:29 -0700 Received: from localhost ([127.0.0.1] helo=talk.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1FlOhI-0002rm-Jw for activemq-users@geronimo.apache.org; Wed, 31 May 2006 04:17:08 -0700 Message-ID: <4642576.post@talk.nabble.com> Date: Wed, 31 May 2006 04:17:08 -0700 (PDT) From: Attila_Szegedi To: activemq-users@geronimo.apache.org Subject: Re: Broker Hung With Following VM Dump In-Reply-To: <4446559.post@talk.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-Sender: nabble2@szegedi.org X-Nabble-From: Attila_Szegedi References: <4409496.post@talk.nabble.com> <4425279.post@talk.nabble.com> <4446559.post@talk.nabble.com> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N I'm experiencing the same problem -- I have two queues (A and B). They're serviced with 40 sessions that have consumers with message listeners for both A and B. If I send a bigger batch of messages to A, everything dies down because message listeners for A also send to B. That is, I have 40 sessions that both consume from B and produce to B (sometimes when they get dispatched a message from A). Once B becomes biggish, everything stops as all consumers for B are also producers for it, and their threads get blocked. Rather unpleasant. Also, whenever I create a producer, I explicitly call producer.setDeliveryMode(DeliveryMode.PERSISTENT); so these should be all persistent queues. Regardless, I see my threads on the client being blocked at "session Task" daemon prio=7 tid=0x0052f1c0 nid=0x1967e00 in Object.wait() [f0f0d000..f0f0eaa0] at java.lang.Object.wait(Native Method) - waiting on <0x61011860> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at java.lang.Object.wait(Object.java:429) at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75) - locked <0x61011860> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318) at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:38) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:70) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1108) at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1524) at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:462) at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:356) and what's even more interesting also at "Thread-2" prio=5 tid=0x0052ead0 nid=0x1a43200 in Object.wait() [f0e8d000..f0e8daa0] at java.lang.Object.wait(Native Method) - waiting on <0x623a2828> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at java.lang.Object.wait(Object.java:429) at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75) - locked <0x623a2828> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318) at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:38) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:70) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1108) at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1636) at org.apache.activemq.ActiveMQMessageConsumer.(ActiveMQMessageConsumer.java:186) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:812) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:772) that is, in consumer creation code. Now, regardless of how loaded the server believes it is, it seems like a bad idea to me to prevent it from creating new consumers that could provide some relief... The server threads are blocked in "tcp:///127.0.0.1:55094" daemon prio=9 tid=0x0055fba0 nid=0x1bb5200 in Object.wait() [f1921000..f1922aa0] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.activemq.memory.UsageManager.waitForSpace(UsageManager.java:85) - locked <0x4c9932f8> (a java.lang.Object) at org.apache.activemq.memory.UsageManager.waitForSpace(UsageManager.java:82) at org.apache.activemq.broker.region.Queue.send(Queue.java:248) at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:195) at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:320) at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:192) at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:109) at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:97) at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:121) at org.apache.activemq.broker.AbstractConnection.processMessage(AbstractConnection.java:346) at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:590) at org.apache.activemq.broker.AbstractConnection.service(AbstractConnection.java:196) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:62) at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:93) at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:70) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:114) at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:87) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:139) at java.lang.Thread.run(Thread.java:552) For the record, all I got is one queue ("A") with 5469 messages (each of them a text message of few hundred bytes at most), one with 223 messages (20-30k serialized object in each), one with 1328 messages ("B") (1-2K), and one with 46 messages (few k each). On JMX console MemoryPercentageUsed of each queue is reported to be 0. Using ActiveMQ-4.0RC3 on Mac OS X 10.4.6,in out-of-the box configuration (just running bin/activemq) -- View this message in context: http://www.nabble.com/Broker+Hung+With+Following+VM+Dump-t1627677.html#a4642576 Sent from the ActiveMQ - User forum at Nabble.com.