From users-return-25600-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Tue Oct 05 18:42:55 2010 Return-Path: Delivered-To: apmail-activemq-users-archive@www.apache.org Received: (qmail 88143 invoked from network); 5 Oct 2010 18:42:55 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 5 Oct 2010 18:42:55 -0000 Received: (qmail 18191 invoked by uid 500); 5 Oct 2010 18:42:55 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 18139 invoked by uid 500); 5 Oct 2010 18:42:54 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 18131 invoked by uid 99); 5 Oct 2010 18:42:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:42:54 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of slewis@fusesource.com designates 209.85.215.171 as permitted sender) Received: from [209.85.215.171] (HELO mail-ey0-f171.google.com) (209.85.215.171) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:42:47 +0000 Received: by eyg7 with SMTP id 7so4045958eyg.2 for ; Tue, 05 Oct 2010 11:42:27 -0700 (PDT) Received: by 10.213.16.144 with SMTP id o16mr1421613eba.64.1286304147205; Tue, 05 Oct 2010 11:42:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.31.140 with HTTP; Tue, 5 Oct 2010 11:42:05 -0700 (PDT) In-Reply-To: <4CAB6A62.6070600@nwea.org> References: <4CA131EE.8090306@nwea.org> <4CA22925.4050707@nwea.org> <4CA36489.5030003@nwea.org> <4CA65D18.2030707@nwea.org> <4CAB6A62.6070600@nwea.org> From: Stan Lewis Date: Tue, 5 Oct 2010 14:42:05 -0400 Message-ID: Subject: Re: producer flow control and message cursors To: Joe Niski , users@activemq.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Sorry Joe, you want to reply to users@activemq.apache.org, looks like my usage of reply all confused things. Looks like you want to increase the temp store system limit, not sure off the top of my head what the default is, but it's probably similar to the commented out setting in the default activemq.xml. With these settings I can put 1M messages onto a queue with no consumer using the default 512MB heap limit: " producerFlowControl=3D"true" memoryLimit=3D"1mb"/> " producerFlowControl=3D"true" memoryLimit=3D"1mb"/> =09 When using the default store based cursor (and KahaDB for your persistence adapter) the tempUsage limit tracks the size of data/localhost/tmp_storage, though not sure if it just looks at the one datafile or includes the whole directory, using the example producer to push 1M messages gives me a temp store size of 5.5G. In general flow control prevents some limit being reached, be it the memory limit on a destination, the overall system memory limit or the store/temp filesize limits, in a way that you wouldn't perhaps need to address in your producer, i.e. if the limit is reached, your producer blocks or slows down a bit, perhaps giving a consumer time to drain the destination that the producer is sending messages to. Message cursors are another way to avoid high memory usage in the broker, by spooling messages someplace other than memory (unless of course you're using the vmcursor) as they're received. In your case I suspect the cursor is doing it's job but you have to increase the temp size limit, as that's what kicks in when you're pushing non-persistent messages. If you're pushing persistent messages then it'd be the store size limit. Hope that helps! On Tue, Oct 5, 2010 at 2:11 PM, Joe Niski wrote: > Thanks, Stan, that was a good suggestion. > > We set it to 1MB. Additional background - for testing purposes, we've set > the JVM heap to 256MB (it's=C2=A0 1.5GB in our UAT and Production environ= ments, > 32-bit Sun jvm). We're using Kahadb for our message store. > > Our test messages are roughly 1kB in size. We ran tests with no consumer = on > the queue, in order to force the messageCursor into working. We monitored > ActiveMQ with the web console and jconsole, in addition to tailing the lo= g > files for the producer and ActiveMQ. > > With producerFlowControl OFF and fileQueueCursor enabled, we could send a > bit over 24,000 messages to the queue, at which point the jvm heap was > exhausted and the producer could no longer connect. > > With producerFlowControl ON and fileQueueCursor enabled, we could send 10= 5 > messages, at which point producerFlowControl blocked the producer. > Activating a consumer on the queue got things flowing, as expected. > > But this question remains: > 1) Is it possible to use messageCursors (either store-based or file-based= ) > in concert with producerFlowControl, so producers can continue even when = the > queue's memoryLimit is reached? > > And i have these new questions: > 2) Is the memoryLimit for queues and topics applied only when > producerFlowControl is enabled? > 3) If the answer to (2) is "no", then why do i see ActiveMQ using up all > available heap when i configure the fileQueueCursor? > 4) Are messageCursors intended to work when producerFlowControl is enable= d > and the queue memoryLimit is reached? > 5) If the answer to (4) is "yes", then why do i see producers blocked whe= n > the memoryLimit is reached? > > The online documentation is ambiguous about how messageCursors and > producerFlowControl are supposed to work together. In the latest release = of > "ActiveMQ in Action",=C2=A0 messageCursors aren't even mentioned. > > Thanks in advance, > Joe > > Joe Niski > IS Development=C2=A0| =C2=A0NWEA > > PHONE 503.212.3382 =C2=A0|=C2=A0 FAX 503.639.7873 > > NWEA.ORG =C2=A0|=C2=A0 Partnering to Help All Kids Learn > > On 10/01/2010 03:18 PM, Stan Lewis wrote: > > It could be that you're hitting the system memory usage limit.=C2=A0 Try > setting a lower memory limit on your queue like 1mb with the > pendingQueuePolicy. > > On Fri, Oct 1, 2010 at 6:13 PM, Joe Niski wrote: >> Well, i may have spoken too soon. Tests with a queue's destination polic= y >> configured like this: >> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0" producerFlowControl=3D"true" >> memoryLimit=3D"64mb"> >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 >> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 >> >> indicate that the pendingQueuePolicy is ignored if producerFlowControl i= s >> enabled. The same test (sending 10,000 messages to a queue that has no >> consumers) locks up at the same message count, regardless of whether the= re's >> a pendingQueuePolicy or not, with this error in the log: >> 2010-10-01 09:39:51,081 | INFO =C2=A0| Kaha Store using data directory >> /opt/activemq/data/localhost/tmp_storage | >> org.apache.activemq.kaha.impl.KahaStore | >> default:memory:queue://inbound.tt2etl.q:memory Usage Thread Pool >> 2010-10-01 09:41:04,855 | INFO =C2=A0| Usage Manager memory limit reache= d on >> queue://inbound.tt2etl.q. Producers will be throttled to the rate at whi= ch >> messages are removed from this destination to prevent flooding it. See >> http://activemq.apache.org/producer-flow-control.html for more info | >> org.apache.activemq.broker.region.Queue | ActiveMQ Transport: >> tcp:///192.168.22.78:52741 >> >> Setting producerFlowControl=3D"false" changes the behavior - messages >> continue to be accepted by the queue, until a consumer starts or AMQ >> actually runs out of heap and the producer's connection is broken.