activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Memory limit errors
Date Wed, 02 Jan 2013 20:53:17 GMT
Yah, it's probably best to figure out how things are being sent right now.

If messages are sent async and without a producer window then you'll hit
the type of flow control that you did.

Messages are sent async by default if using non-persistent messages or
during a transaction. The only time any synchronous communication will be
done during a transaction is when you commit (or rollback).

So you can set ActiveMQConnection#setAlwaysSyncSend which will always send
sync for all types of sends (non-persistent and during a transaction). This
will enable the type of flow control on the broker side that blocks only
the one producer. The drawbacks are it has more network overhead and is
likely slower. How much slower is something you may wish to measure.
Depending on your application requirements, it might not be a big deal.

You can also set the producer window on the client. The producer window
establishes a client-side contract that basically says "I will only sned N
number of bytes to you until I hear back that the message has been
received" You can set this value (in number of bytes) on
ActiveMQConnection#setProducerWindowSize. This will also achieve blocking
only the one producer instead of the entire connection.

Lastly, note that persistent messages are always sent synchronous so by
default would block only the producer and not the entire connection.


On Wed, Jan 2, 2013 at 1:40 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:

> Thanks! Sounds like there might be another way we can configure the flow
> control that blocks only one producer? Do you have any recommendation on
> what changes I should make to the config?
>
> On Wed, Jan 2, 2013 at 12:33 PM, Christian Posta
> <christian.posta@gmail.com>wrote:
>
> > Enqueue count = number of messages sent to the queue
> > Dequeue count = number of messages ack'd by the consumer
> > Inflight count = number of messages sent to consumer's session (and are
> not
> > ack'd by consumer)
> > Dispatch count = messages sent to the consumer (dequeue + inflight)
> >
> > I asked about the async sends because the producer flow control you hit
> was
> > the kind that will block the entire connection (instead of just blocking
> > that one producer).
> >
> >
> >
> >
> >
> > On Wed, Jan 2, 2013 at 12:41 PM, Mohit Anchlia <mohitanchlia@gmail.com
> > >wrote:
> >
> > > Thanks again! Is the dispatch queue size the one that relates to what's
> > > held in memory?
> > >
> > > We are using mule and the endpoint has no transactions configured. What
> > did
> > > you mean by forcing async?
> > >
> > > On Wed, Jan 2, 2013 at 11:06 AM, Christian Posta
> > > <christian.posta@gmail.com>wrote:
> > >
> > > > Correct. The memoryUsage is increased when messages are held in
> memory.
> > > > This happens when messages are paged in from the store. They are then
> > > > dispatched to consumers and held until the consumer acks.
> > > >
> > > > In your case looks like a particular destination (eventsEndpoint) is
> > > > filling up somehow. Can you look in JMX and see how many enqueued,
> > > > dispatched, dequeued, inflight etc? Also check to see how many
> > > > subscriptions to that destination, and see what those
> dispatch-counter
> > > and
> > > > dispatch queue size looks like.
> > > >
> > > > BTW any chance you're sending (producers) using a transaction, or
> > forcing
> > > > sends to be async?
> > > >
> > > >
> > > >
> > > > On Wed, Jan 2, 2013 at 11:51 AM, Mohit Anchlia <
> mohitanchlia@gmail.com
> > > > >wrote:
> > > >
> > > > > So if I understand correctly I shouldn't be seeing this behaviour
> > > unless
> > > > > something is being held in memory. We ack immediately so I am kind
> of
> > > > > wondering what might be going on here. Is there a way to find out
> > what
> > > > > might be happening here?
> > > > >
> > > > > On Wed, Jan 2, 2013 at 10:03 AM, Christian Posta
> > > > > <christian.posta@gmail.com>wrote:
> > > > >
> > > > > > What info can you provide about how clients are consuming from
> the
> > > > > queues?
> > > > > > Any chance there are a bunch of messages "inflight" which haven't
> > > been
> > > > > > ack'd by the consumer?
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 2, 2013 at 10:20 AM, Mohit Anchlia <
> > > mohitanchlia@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > I see these messages being logged. Queues are setup to
use
> > > persistent
> > > > > > > store. I am wondering what's causing this messages to get
> > blocked.
> > > I
> > > > > see
> > > > > > > there is a memoryLimit but if I am using persistent=true
then
> > > should
> > > > > this
> > > > > > > be the desired behaviour?
> > > > > > >
> > > > > > >
> > > > > > > 2013-01-02 05:06:13,388 | INFO |
> > > > > > > Usage(default:memory:queue://eventsEndpoint:memory)
> > > percentUsage=0%,
> > > > > > > usage=0, limit=20971520,
> > > > > > > percentUsageMinDelta=1%;Parent:Usage(default:memory)
> > > > percentUsage=106%,
> > > > > > > usage=22366029, limit=20971520, percentUsageMinDelta=1%:
Usage
> > > > Manager
> > > > > > > Memory Limit reached. Producer
> > > > > > > (ID:pfdamq301.ie.net-54582-1351809386355-2:1:1:1) stopped
to
> > > prevent
> > > > > > > flooding queue://eventsEndpoint. See
> > > > > > >
> > > > > > >
> > > > > > >    <broker xmlns="http://activemq.apache.org/schema/core"
> > > > > > > brokerName="static-broker1" persistent="true"
> > > > > > > dataDirectory="${activemq.data}">
> > > > > > >
> > > > > > >         <!-- Destination specific policies using destination
> > names
> > > or
> > > > > > > wildcards -->
> > > > > > >         <destinationPolicy>
> > > > > > >             <policyMap>
> > > > > > >                 <policyEntries>
> > > > > > >                     <policyEntry queue=">"
> > > producerFlowControl="true"
> > > > > > > memoryLimit="20mb">
> > > > > > >                         <deadLetterStrategy>
> > > > > > >                           <individualDeadLetterStrategy
> > > > > > queuePrefix="DLQ."
> > > > > > > useQueueForQueueMessages="true" />
> > > > > > >                         </deadLetterStrategy>
> > > > > > >                     </policyEntry>
> > > > > > >                     <policyEntry topic=">"
> > > producerFlowControl="true"
> > > > > > > memoryLimit="20mb">
> > > > > > >                     </policyEntry>
> > > > > > >                 </policyEntries>
> > > > > > >             </policyMap>
> > > > > > >         </destinationPolicy>
> > > > > > >
> > > > > > >         <!-- Use the following to configure how ActiveMQ
is
> > exposed
> > > > in
> > > > > > JMX
> > > > > > > -->
> > > > > > >         <managementContext>
> > > > > > >             <managementContext createConnector="true"/>
> > > > > > >         </managementContext>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Christian Posta*
> > > > > > http://www.christianposta.com/blog
> > > > > > twitter: @christianposta
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Christian Posta*
> > > > http://www.christianposta.com/blog
> > > > twitter: @christianposta
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message