activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Mielke <>
Subject Re: Producer flow control
Date Wed, 15 Feb 2012 09:50:33 GMT

> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.

Yes you can. However when using persistent queue messages, producerFlowControl does not kick-in
that early. Which is actually a good thing in general.

All this is configurable as well. You can specify cursorMemoryHighWaterMark="100" on your
destination policy configuration. With that the cursor will take up to 100% of the configured
queue memory limit. When that limit is reached, producer flow control will also kick in (as
now 100% of the memory are in use). 

> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?

ProducerFlowControl and perhaps in your case also setting cursorMemoryHighWaterMark="100"
on the destination policy.

Best Regards,

Torsten Mielke

On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:

> Hi,
> thanks for the answer.
> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.
> If i have a consumer which is able to process only 10 messages by
> seconds and my queue is filled with 36000 messages (due to faster
> producers), i know that it will take 1 hour to process all my
> messages. I would reject new messages as my processing time shall be
> less than one hour.
> I would also keep the order of incoming messages (small and big) so i
> can't easily manage two queues without adding a complex
> synchronization mechanism.
> I see for
> non-persistent messages but it should not apply for persistent ones.
> (Prefetch is configurable for persistence :
> but nothing for Pending Message Limit Strategy).
> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?
> Modifying the cursor, producer flow control, another thing ?
> Regards
> Hervé
> On 2/14/12, Torsten Mielke <> wrote:
>> Hello Herve,
>> You cannot configure producer flow control based on the various message
>> sizes.
>> However you could consider using different queues for these types of msgs
>> and configure
>> the limits for each queue differently. That way your small msgs won't be
>> blocked just because of a small amount of large msgs on the same queue
>> (assuming different producers for both kinds of msgs).
>> Also, when sending persistent queue messages, you rarely see producer flow
>> control happening when using the default store cursor. That is because the
>> store cursor is configured to hold messages in memory up to only 70% of the
>> configured destination limit. If this 70% limit is reached, it won't keep
>> new msgs in memory but reload them from the store (kahadb) later (when its
>> time to dispatch the msg). Remember a persistent msg is first written to the
>> store when received by the broker.
>> Producer flow control however kicks in when 100% of the available memory for
>> each destination are used. This limit however you generally don't reach.
>> Things are different for non-persistent msgs. The default file cursor keeps
>> msgs in memory until 100% of the configured destination limit are reached.
>> Thereafter it swaps msgs to temp storage (which is different from kahadb)
>> and producer flow control would kick in.
>> Hope this helps.
>> Torsten Mielke
>> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>>> Hi,
>>> i have looked to the documentation of the producer flow control
>>> functionality (
>>> I see that it is based on the storage size but in my case, i think not
>>> being able to use the size.
>>> I have a queue where i put messages two kind of messages : Big 10MB and
>>> Small 1kB.
>>> As Big messages are rare (compared to small ones), i would limit the
>>> Memory
>>> to 100MB (10 messages).
>>> But for small messages i would limit the number of small messages to 5000
>>> (so about 5MB) . We have noted that with JDBC persistence, the number of
>>> pending messages in the broker (not by queue) is important for
>>> "performances".
>>> As i would keep the order of Big and Small messages, i put them in the
>>> same
>>> queue.
>>> Is there a way to define a strategy for the flow control which is 100MB
>>> and
>>> 5k messages ?
>>> Regards
>>> Hervé

View raw message