activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hervé BARRAULT <>
Subject Re: Producer flow control
Date Tue, 14 Feb 2012 13:20:08 GMT
thanks for the answer.

So when using persistence, i can't rely on producerFlowControl to
limit the Flow.

If i have a consumer which is able to process only 10 messages by
seconds and my queue is filled with 36000 messages (due to faster
producers), i know that it will take 1 hour to process all my
messages. I would reject new messages as my processing time shall be
less than one hour.

I would also keep the order of incoming messages (small and big) so i
can't easily manage two queues without adding a complex
synchronization mechanism.

I see for
non-persistent messages but it should not apply for persistent ones.
(Prefetch is configurable for persistence :
but nothing for Pending Message Limit Strategy).

What is the best way to add a mechanism to constrain the queue size
(handling slow consumer when using persistence) ?

Modifying the cursor, producer flow control, another thing ?



On 2/14/12, Torsten Mielke <> wrote:
> Hello Herve,
> You cannot configure producer flow control based on the various message
> sizes.
> However you could consider using different queues for these types of msgs
> and configure
> the limits for each queue differently. That way your small msgs won't be
> blocked just because of a small amount of large msgs on the same queue
> (assuming different producers for both kinds of msgs).
> Also, when sending persistent queue messages, you rarely see producer flow
> control happening when using the default store cursor. That is because the
> store cursor is configured to hold messages in memory up to only 70% of the
> configured destination limit. If this 70% limit is reached, it won't keep
> new msgs in memory but reload them from the store (kahadb) later (when its
> time to dispatch the msg). Remember a persistent msg is first written to the
> store when received by the broker.
> Producer flow control however kicks in when 100% of the available memory for
> each destination are used. This limit however you generally don't reach.
> Things are different for non-persistent msgs. The default file cursor keeps
> msgs in memory until 100% of the configured destination limit are reached.
> Thereafter it swaps msgs to temp storage (which is different from kahadb)
> and producer flow control would kick in.
> Hope this helps.
> Torsten Mielke
> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>> Hi,
>> i have looked to the documentation of the producer flow control
>> functionality (
>> I see that it is based on the storage size but in my case, i think not
>> being able to use the size.
>> I have a queue where i put messages two kind of messages : Big 10MB and
>> Small 1kB.
>> As Big messages are rare (compared to small ones), i would limit the
>> Memory
>> to 100MB (10 messages).
>> But for small messages i would limit the number of small messages to 5000
>> (so about 5MB) . We have noted that with JDBC persistence, the number of
>> pending messages in the broker (not by queue) is important for
>> "performances".
>> As i would keep the order of Big and Small messages, i put them in the
>> same
>> queue.
>> Is there a way to define a strategy for the flow control which is 100MB
>> and
>> 5k messages ?
>> Regards
>> Hervé

View raw message