Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C9B4F10EA4 for ; Wed, 7 Aug 2013 02:16:44 +0000 (UTC) Received: (qmail 79363 invoked by uid 500); 7 Aug 2013 02:16:44 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 79330 invoked by uid 500); 7 Aug 2013 02:16:44 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 79322 invoked by uid 99); 7 Aug 2013 02:16:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 02:16:44 +0000 X-ASF-Spam-Status: No, hits=2.0 required=5.0 tests=SPF_NEUTRAL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: encountered temporary error during SPF processing of domain of jm15119b@gmail.com) Received: from [216.139.250.139] (HELO joe.nabble.com) (216.139.250.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 02:16:37 +0000 Received: from [192.168.236.139] (helo=joe.nabble.com) by joe.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1V6tHf-0001sm-Jg for users@activemq.apache.org; Tue, 06 Aug 2013 19:15:31 -0700 Date: Tue, 6 Aug 2013 19:15:16 -0700 (PDT) From: Jmal To: users@activemq.apache.org Message-ID: <1375841716589-4670137.post@n4.nabble.com> In-Reply-To: References: <1375403257331-4670034.post@n4.nabble.com> <1375736745634-4670103.post@n4.nabble.com> Subject: Re: Help with a producer & consumer stall problem after reaching flow control MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Yup, it turns out there was a logic error in app2 causing this problem. As for app1 hitting producer flow control, it's not normally expected, but it was a frequent occurrence recently given the increased load on app2. Thanks for your help. ceposta wrote > You may want to also look at app2 and see what it's doing and why its not > receiving or ack'ing messages. Is the app1 hitting producer flow control > expected? > > > On Mon, Aug 5, 2013 at 2:05 PM, Jmal < > jm15119b@ > > wrote: > >> I placed the broker in debug mode, and did not notice any log messages >> pertaining to reaching the limit, and/or the broker waiting for consumer >> responses. Would DEBUG log level output a particular message(s) giving a >> clue about this problem? >> >> Thanks for the help. >> >> ceposta wrote >> > You'll have to check to see what the broker thinks is going on. So if >> the >> > broker dispatches messages up to the consumer prefetch limit, and the >> > consumer hasn't sent ack's back, the broker will not try to send any >> more >> > messages and your consumers will look hung. >> > >> > >> > On Thu, Aug 1, 2013 at 5:27 PM, Jmal < >> >> > jm15119b@ >> >> > > wrote: >> > >> >> Hello, I'm hoping that someone would be able to help me understand why >> I >> >> am >> >> having an issue with my applications usage of ActiveMQ under flow >> control >> >> situations. I'm using 2 ActiveMQ 5.4.3 brokers using Open JDK 1.6, >> >> configured with 5 queues each. I have 2 applications in separate JVMs >> >> connecting to each broker independently (app1 connects to broker1 & >> >> broker2) >> >> and (app2 connects to broker2). app1 pushes messages to 1 or more >> queues >> >> on >> >> broker2 using the broker2's nio URI. app2 reads and processes those >> >> messages on broker2. All works well, until broker's 1 & 2 memory >> usage >> >> reaches 100% (as observed by the admin UI). Then app1's connection to >> >> broker2 is placed in flow control as expected. The weird part is that >> >> app2 >> >> is no longer able to read any messages from broker2 which is very >> >> unexpected. If I understand correctly, producer flow control is only >> >> suppose to slow down the producers from app1's connection to broker2, >> and >> >> not app2's connection to broker2? If this is true, is there a known >> bug >> >> with the version of ActiveMQ I'm using or have I mis-configured the >> >> broker? >> >> Below is the config used for both brokers, and each is running on 2 >> >> separate >> >> EC2 instances with 7GB of memory. Any help will be greatly >> appreciated. >> >> >> >> Thanks >> >> >> > > > > >> >> > xmlns="http://www.springframework.org/schema/beans" >> >> xmlns:amq="http://activemq.apache.org/schema/core" >> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> >> http://activemq.apache.org/schema/core >> >> http://activemq.apache.org/schema/core/activemq-core.xsd"> >> >> >> >> >> > > > > >> >> > xmlns="http://activemq.apache.org/schema/core" >> >> brokerName="localhost" >> >> dataDirectory="/usr/data/activemq" >> >> destroyApplicationContextOnStop="true" >> >> persistent="true"> >> >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >> >> > > >> >> >> > > > > >> >> > /> >> >> >> > > >> >> >> >> >> > > >> >> >> > > > > >> >> > directory="/usr/data/activemq/kahadb" >> >> enableJournalDiskSyncs="false" >> >> indexWriteBatchSize="10000" >> >> indexCacheSize="1000" >> >> /> >> >> >> > > >> >> >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >> >> > > >> >> >> > > > > >> >> > trustStore="truststore" trustStorePassword="efgh5678"/> >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://activemq.2283324.n4.nabble.com/Help-with-a-producer-consumer-stall-problem-after-reaching-flow-control-tp4670034.html >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >> > >> > >> > >> > -- >> > *Christian Posta* >> > http://www.christianposta.com/blog >> > twitter: @christianposta >> >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/Help-with-a-producer-consumer-stall-problem-after-reaching-flow-control-tp4670034p4670103.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta -- View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-producer-consumer-stall-problem-after-reaching-flow-control-tp4670034p4670137.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.