Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 679 invoked from network); 2 Sep 2009 19:39:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 2 Sep 2009 19:39:20 -0000 Received: (qmail 72801 invoked by uid 500); 2 Sep 2009 19:39:20 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 72751 invoked by uid 500); 2 Sep 2009 19:39:20 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 72741 invoked by uid 99); 2 Sep 2009 19:39:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 19:39:19 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [81.169.137.254] (HELO h57067.serverkompetenz.net) (81.169.137.254) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 19:39:09 +0000 Received: from h57067.serverkompetenz.net (unknown [127.0.0.1]) by h57067.serverkompetenz.net (Postfix) with ESMTP id 27622802F0 for ; Wed, 2 Sep 2009 19:38:47 +0000 (UTC) Received: from [192.168.49.120] (unknown [213.160.26.250]) by h57067.serverkompetenz.net (Postfix) with ESMTP for ; Wed, 2 Sep 2009 19:38:47 +0000 (UTC) Message-Id: <083EBE96-D904-40DF-B58D-9147A0D83335@soa-knowledge.net> From: Andreas Gies To: users@activemq.apache.org In-Reply-To: <87A0FB01-DA92-46E6-8A4C-4A428E53E4A5@gmail.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: Producers hang under load with AMQ 5.2 Date: Wed, 2 Sep 2009 21:38:38 +0200 References: <25251841.post@talk.nabble.com> <64262657-34DF-4F82-9E81-A85D9993E57A@soa-knowledge.net> <25255812.post@talk.nabble.com> <87A0FB01-DA92-46E6-8A4C-4A428E53E4A5@gmail.com> X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org Hi again, if it turns out to be a consumer, you could use the concept of a Virtual topics to have several groups of consumers, each group getting a message copy, but the group members sharing the load. To your question, You need to decide on a strategy what you do in case of slow consumers. I think you are experiencing the default - flow control. See Rob's suggestion as well. Besides you *could* configure the broker in a way that the send throws an exception if the situation occurs, but then that exception must be handled on the producer site. Best regards Andreas On Sep 2, 2009, at 2:36 PM, Rob Davies wrote: > Hi Tanel, > > It would be worth disabling flow control - e.g. > > disable flow control by setting the producerFlowControl flag to > false on a destination policy in the Broker configuration - e.g. > > > > > memoryLimit="1mb"> > > > > > > > > > > > > On 2 Sep 2009, at 12:35, wo wrote: > >> >> Hi. >> >> We have 1 producer app and 2 consumers in total. Messages are >> distributed >> evenly between 2 separate topics. >> Slow consumer should not be a problem, both consumers use >> java.util.concurrent Executors working off an unbounded queue to >> process the >> incoming messages and every attempt has been made to make jms message >> handoff to executors as fast as possible. >> >> Messages itself are small, maybe 100 bytes of data in each. Our >> topic memory >> limit in activemq config is set to 20mb which seems sensible >> enough. Would >> it help to increase it further? >> >> I'll try to confirm whether the consumer is to blame but even if it >> was, is >> there any workaround besides not putting a load on the system? :) >> I'd say >> its a pretty major issue with activemq, one slow consumer shouldn't >> be able >> to bring the whole thing down. >> >> Regards, >> Tanel >> >> >> Andreas Gies wrote: >>> >>> Hi there, >>> >>> I am wondering how many consumers there are on the topic and if >>> there >>> are any slow consumers that >>> can't keep up with the messaging load. Without having verified it, >>> it >>> seems that your app hangs in the >>> the send message waiting for the response from the broker. That >>> would >>> happen, if the sending application >>> can't place the message in the broker. >>> >>> One thing strikes me odd....if all of your consumers keep consuming, >>> that shouldn't happen. So it might >>> be a starting point to double check them. >>> >>> As for the Advisory messages: Yes, thats normal. For more details on >>> advisories see >>> http://activemq.apache.org/advisory-message.html. You *could turn >>> advisories off, >>> but you need to remember to reenable them if you are switching to a >>> network of brokers >>> some day ;) >>> >>> Hope that helps a bit >>> Andreas >>> >>> >>> On Sep 2, 2009, at 7:24 AM, wo wrote: >>> >>>> >>>> Hi. >>>> >>>> We're having a problem under moderate to heavy load (~2500 >>>> nonpersisted >>>> messages /second on a single broker) while producers will just hang >>>> indefinitely at some point and producer application needs to be >>>> restarted in >>>> order to continue. >>>> >>>> Here's a snippet from a producer application thread dump after >>>> hang: >>>> >>>> at >>>> java >>>> .util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java: >>>> 317) >>>> at >>>> org >>>> .apache >>>> .activemq.transport.FutureResponse.getResult(FutureResponse.java: >>>> 40) >>>> at >>>> org >>>> .apache >>>> .activemq >>>> .transport.ResponseCorrelator.request(ResponseCorrelator.java:80) >>>> at >>>> org >>>> .apache >>>> .activemq >>>> .ActiveMQConnection.syncSendPacket(ActiveMQConnection.java: >>>> 1195) >>>> at >>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1644) >>>> - locked <0x00002aaab5abeb30> (a java.lang.Object) >>>> at >>>> org >>>> .apache >>>> .activemq >>>> .ActiveMQMessageProducer.send(ActiveMQMessageProducer.java: >>>> 227) >>>> at >>>> org >>>> .apache >>>> .activemq >>>> .ActiveMQMessageProducerSupport >>>> .send(ActiveMQMessageProducerSupport.java:241) >>>> at >>>> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java: >>>> 597) >>>> >>>> So far we've tried upgrading AMQ from 5.1 to 5.2 and increasing the >>>> memory >>>> limits in broker config, both giving no result, the problem still >>>> exists. >>>> Neither application nor activemq log displays any errors / >>>> exceptions. >>>> >>>> One thing i have noticed is that for every message sent to a topic, >>>> there >>>> are 2 advisory messages being produced in >>>> ActiveMQ.Advisory.Producer.Topic.TOPICNAME which strikes me as odd >>>> but i >>>> haven't been able to find any information in forums or manual about >>>> whether >>>> this is normal behaviour? >>>> >>>> Producer side is a standalone Spring application with AMQ >>>> connectionfactory >>>> wrapped in spring SingleConnectionFactory and used in jmstemplate, >>>> as per >>>> example. >>>> >>>> Any insight would be greatly appreciated. >>>> >>>> Tanel Unt >>>> Core Developer >>>> Tracking Center Ltd. >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25251841.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >>> >>> >>> >> >> -- >> View this message in context: http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25255812.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > Rob Davies > I work here: http://fusesource.com > My Blog: http://rajdavies.blogspot.com/ > I'm writing this: http://www.manning.com/snyder/ > > > >