Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of gary.tully@gmail.com
 designates 209.85.218.215 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=x12PploV0zMhOG/ooyz4oA66oYCDtBSy5G3K0aTs3JR234rWTPSkHLzgFWSE9MsK0P
         DUa3KSfp9tXGKriMrOiwGfqypEnIsFhCsgagSkd6lF0o4R4SmtZYUo5JKabzIlAb12fh
         Ae5ogSK8hrk9juOkNJZ1/UslVIVQ1Wr55/qAQ=
MIME-Version: 1.0
In-Reply-To: <83405999-EC99-43CE-B7E7-4CF1DE1ECF75@pobox.com>
References: <34356D9E-DC34-4C47-BED6-5B0A06C4C1FF@pobox.com>
	 <3a73c17c0912020138o3290cf21i53880673f76d78dc@mail.gmail.com>
	 <83405999-EC99-43CE-B7E7-4CF1DE1ECF75@pobox.com>
Date: Wed, 2 Dec 2009 11:35:18 +0000
Message-ID: <3a73c17c0912020335t11bc91bbm5760778e1a892390@mail.gmail.com>
Subject: Re: Producer flow control and hard lockup of 5.3.0
From: Gary Tully <gary.tully@gmail.com>
To: users@activemq.apache.org
Content-Type: multipart/alternative; boundary=00151758ac46e5860a0479bd4415

--00151758ac46e5860a0479bd4415
Content-Type: text/plain; charset=ISO-8859-1

stock activemq.xml should be fine. The client ack in the tests is because
the limits are such that only one message can fit in the queue so that the
ack behavior is exactly deterministic w.r.t memory usage.
JMX instrumentation will help. Use jconsole and when the queue is drained,
what does the Broker say w.r.t to memory usage and for the queue w.r.t
cursor memory usage. Both should go to 0. If they are not at zero you may be
experiencing some of the known issues that effect memory management
accounting w.r.t the file based cursor in 5.3.0 that have been resolved on
trunk. If this is the case, I would recommend trying  a run with the latest
5.4-SNAPSHOT and if that shows the problem to progress to opening an issue
and attaching your test case.


2009/12/2 Geir Magnusson Jr. <geir@pobox.com>

> Thanks for the response.
>
> On Dec 2, 2009, at 4:38 AM, Gary Tully wrote:
>
> > can you post your activemq.xml?
>
> It's as distributed in 5.3.0
>
> > possibly compare you setup to what is known to work from the unit tests:
> > org.apache.activemq.ProducerFlowControlTest
> >
> >
> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/ProducerFlowControlTest.java?view=markup
>
>
> Thx.  The only thing that jumps out at me is that
>
> a) I'm not ever calling setUseAsyncSend on the factory, but as my messages
> are non-persistent, I don't know if it matters.  I can try it though,
> although I'm loathe to just "try things" :)
> b) I use auto ack rather than client ack.  Does that make a diff?
>
> I did try OpenMQ with the same code, and everything just worked - ran about
> 10MM messages until I got bored. I would prefer to use ActiveMQ though.  Is
> the fact I have a stock activemq.xml relevant?  Is there any way I can ask
> the broker what it thinks about the state of affairs when this happens?  any
> verbose logging modes beyond the log message I mentioned?
>
> geir
>
>
> >
> > 2009/12/1 Geir Magnusson Jr. <geir@pobox.com>
> >
> >> I'm doing some simple stress experiments and am able to lockup 5.3.0.  I
> >> have a simple producer and consumer as described below.  The queue is
> >> non-perstent, moving small text messages.  Flailing about, I've set my
> >> producer window size to 1024000.
> >>
> >> Upshot is that I can flood the queue for about 250k to 300k messages
> >> overall, and the producer eventually gets stopped.  THe consumer seems
> to
> >> get all of them, at least according to the console which shows 0
> messages in
> >> queue, eventually.
> >>
> >> At this point, the producer never starts again, and even bouncing the
> >> producer doesn't get messages flowing again.  The only thing that works
> is
> >> bouncing the broker, but clearly that's a crappy solution.
> >>
> >> What am I doing wrong?
> >>
> >> geir
> >>
> >>
> >>
> >> Producer - single one, thread...
> >>
> >>       ActiveMQConnectionFactory factory = new
> >> ActiveMQConnectionFactory("tcp://10.0.0.2:61616");
> >>       factory.setProducerWindowSize(1024000);
> >>
> >>       Connection connection = factory.createConnection();
> >>       connection.start();
> >>
> >>       Session session = connection.createSession(false,
> >> Session.AUTO_ACKNOWLEDGE);
> >>       Queue queue = session.createQueue("svc");
> >>       MessageProducer producer = session.createProducer(queue);
> >>       producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
> >>
> >>       while(true) {
> >>
> >>           while(true) {
> >>               TextMessage tm = session.createTextMessage(theMessage);
> >>               producer.send(tm);
> >>           }
> >>       }
> >>
> >> That simply floods the queue with small text messages.
> >>
> >> On the consumer side, I have 15 threads, each of which does something
> like
> >> :
> >>
> >>       ActiveMQConnectionFactory factory = new
> >> ActiveMQConnectionFactory(_brokerURL);
> >>
> >>       _connection = factory.createConnection();
> >>       Session _session = _connection.createSession(false,
> >> Session.AUTO_ACKNOWLEDGE);
> >>
> >>      Queue queue = _session.createQueue(name);
> >>
> >>      MessageConsumer consumer = _session.createConsumer(queue);
> >>
> >>      consumer.setMessageListener(new MessageHandlerAdapter(_session,
> >> msgHandler));
> >>
> >> where my MessageHandlerAdapter has a pretty boring onMessage() method.
> >>
> >> Things go fine until I see the now-dreaded
> >>
> >> INFO | Usage Manager memory limit reached on queue://svc. Producers will
> be
> >> throttled to the rate at which messages are removed from this
> destination to
> >> prevent flooding it. See
> >> http://activemq.apache.org/producer-flow-control.html for more info
> >>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

--00151758ac46e5860a0479bd4415--