activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Mielke <tors...@fusesource.com>
Subject Re: Producer flow control
Date Wed, 15 Feb 2012 09:50:33 GMT
Hello,

> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.


Yes you can. However when using persistent queue messages, producerFlowControl does not kick-in
that early. Which is actually a good thing in general.

All this is configurable as well. You can specify cursorMemoryHighWaterMark="100" on your
destination policy configuration. With that the cursor will take up to 100% of the configured
queue memory limit. When that limit is reached, producer flow control will also kick in (as
now 100% of the memory are in use). 


> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?

ProducerFlowControl and perhaps in your case also setting cursorMemoryHighWaterMark="100"
on the destination policy.


Best Regards,

Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com


On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:

> Hi,
> thanks for the answer.
> 
> So when using persistence, i can't rely on producerFlowControl to
> limit the Flow.
> 
> If i have a consumer which is able to process only 10 messages by
> seconds and my queue is filled with 36000 messages (due to faster
> producers), i know that it will take 1 hour to process all my
> messages. I would reject new messages as my processing time shall be
> less than one hour.
> 
> I would also keep the order of incoming messages (small and big) so i
> can't easily manage two queues without adding a complex
> synchronization mechanism.
> 
> I see http://activemq.apache.org/slow-consumer-handling.html for
> non-persistent messages but it should not apply for persistent ones.
> (Prefetch is configurable for persistence :
> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
> but nothing for Pending Message Limit Strategy).
> 
> What is the best way to add a mechanism to constrain the queue size
> (handling slow consumer when using persistence) ?
> 
> Modifying the cursor, producer flow control, another thing ?
> 
> Regards
> 
> Hervé
> 
> 
> On 2/14/12, Torsten Mielke <torsten@fusesource.com> wrote:
>> Hello Herve,
>> 
>> You cannot configure producer flow control based on the various message
>> sizes.
>> However you could consider using different queues for these types of msgs
>> and configure
>> the limits for each queue differently. That way your small msgs won't be
>> blocked just because of a small amount of large msgs on the same queue
>> (assuming different producers for both kinds of msgs).
>> 
>> Also, when sending persistent queue messages, you rarely see producer flow
>> control happening when using the default store cursor. That is because the
>> store cursor is configured to hold messages in memory up to only 70% of the
>> configured destination limit. If this 70% limit is reached, it won't keep
>> new msgs in memory but reload them from the store (kahadb) later (when its
>> time to dispatch the msg). Remember a persistent msg is first written to the
>> store when received by the broker.
>> Producer flow control however kicks in when 100% of the available memory for
>> each destination are used. This limit however you generally don't reach.
>> Things are different for non-persistent msgs. The default file cursor keeps
>> msgs in memory until 100% of the configured destination limit are reached.
>> Thereafter it swaps msgs to temp storage (which is different from kahadb)
>> and producer flow control would kick in.
>> 
>> Hope this helps.
>> 
>> 
>> Torsten Mielke
>> torsten@fusesource.com
>> tmielke@blogspot.com
>> 
>> 
>> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>> 
>>> Hi,
>>> 
>>> i have looked to the documentation of the producer flow control
>>> functionality (http://activemq.apache.org/producer-flow-control.html).
>>> 
>>> I see that it is based on the storage size but in my case, i think not
>>> being able to use the size.
>>> I have a queue where i put messages two kind of messages : Big 10MB and
>>> Small 1kB.
>>> 
>>> As Big messages are rare (compared to small ones), i would limit the
>>> Memory
>>> to 100MB (10 messages).
>>> But for small messages i would limit the number of small messages to 5000
>>> (so about 5MB) . We have noted that with JDBC persistence, the number of
>>> pending messages in the broker (not by queue) is important for
>>> "performances".
>>> 
>>> As i would keep the order of Big and Small messages, i put them in the
>>> same
>>> queue.
>>> 
>>> Is there a way to define a strategy for the flow control which is 100MB
>>> and
>>> 5k messages ?
>>> 
>>> 
>>> Regards
>>> Hervé





Mime
View raw message