activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Mielke <tors...@fusesource.com>
Subject Re: Producer flow control
Date Mon, 20 Feb 2012 09:25:56 GMT
Hi,

You could use the vmQueueCursor as well. 
In both cases (assuming producerFlowControl=true) the cursors will only dispatch msg from
memory and not from the store. 

Regards,
Torsten Mielke


On Feb 15, 2012, at 3:48 PM, Hervé BARRAULT wrote:

> Hi,
> If i set cursorMemoryHighWaterMark to 100, is it a good idea to use
> vmCursor instead of File Based Cursor (the default one) ?
> 
> Regards
> Hervé
> 
> On Wed, Feb 15, 2012 at 10:50 AM, Torsten Mielke <torsten@fusesource.com>wrote:
> 
>> Hello,
>> 
>>> So when using persistence, i can't rely on producerFlowControl to
>>> limit the Flow.
>> 
>> 
>> Yes you can. However when using persistent queue messages,
>> producerFlowControl does not kick-in that early. Which is actually a good
>> thing in general.
>> 
>> All this is configurable as well. You can specify
>> cursorMemoryHighWaterMark="100" on your destination policy configuration.
>> With that the cursor will take up to 100% of the configured queue memory
>> limit. When that limit is reached, producer flow control will also kick in
>> (as now 100% of the memory are in use).
>> 
>> 
>>> What is the best way to add a mechanism to constrain the queue size
>>> (handling slow consumer when using persistence) ?
>> 
>> ProducerFlowControl and perhaps in your case also setting
>> cursorMemoryHighWaterMark="100" on the destination policy.
>> 
>> 
>> Best Regards,
>> 
>> Torsten Mielke
>> torsten@fusesource.com
>> tmielke@blogspot.com
>> 
>> 
>> On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
>> 
>>> Hi,
>>> thanks for the answer.
>>> 
>>> So when using persistence, i can't rely on producerFlowControl to
>>> limit the Flow.
>>> 
>>> If i have a consumer which is able to process only 10 messages by
>>> seconds and my queue is filled with 36000 messages (due to faster
>>> producers), i know that it will take 1 hour to process all my
>>> messages. I would reject new messages as my processing time shall be
>>> less than one hour.
>>> 
>>> I would also keep the order of incoming messages (small and big) so i
>>> can't easily manage two queues without adding a complex
>>> synchronization mechanism.
>>> 
>>> I see http://activemq.apache.org/slow-consumer-handling.html for
>>> non-persistent messages but it should not apply for persistent ones.
>>> (Prefetch is configurable for persistence :
>>> 
>> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
>>> but nothing for Pending Message Limit Strategy).
>>> 
>>> What is the best way to add a mechanism to constrain the queue size
>>> (handling slow consumer when using persistence) ?
>>> 
>>> Modifying the cursor, producer flow control, another thing ?
>>> 
>>> Regards
>>> 
>>> Hervé
>>> 
>>> 
>>> On 2/14/12, Torsten Mielke <torsten@fusesource.com> wrote:
>>>> Hello Herve,
>>>> 
>>>> You cannot configure producer flow control based on the various message
>>>> sizes.
>>>> However you could consider using different queues for these types of
>> msgs
>>>> and configure
>>>> the limits for each queue differently. That way your small msgs won't be
>>>> blocked just because of a small amount of large msgs on the same queue
>>>> (assuming different producers for both kinds of msgs).
>>>> 
>>>> Also, when sending persistent queue messages, you rarely see producer
>> flow
>>>> control happening when using the default store cursor. That is because
>> the
>>>> store cursor is configured to hold messages in memory up to only 70% of
>> the
>>>> configured destination limit. If this 70% limit is reached, it won't
>> keep
>>>> new msgs in memory but reload them from the store (kahadb) later (when
>> its
>>>> time to dispatch the msg). Remember a persistent msg is first written
>> to the
>>>> store when received by the broker.
>>>> Producer flow control however kicks in when 100% of the available
>> memory for
>>>> each destination are used. This limit however you generally don't reach.
>>>> Things are different for non-persistent msgs. The default file cursor
>> keeps
>>>> msgs in memory until 100% of the configured destination limit are
>> reached.
>>>> Thereafter it swaps msgs to temp storage (which is different from
>> kahadb)
>>>> and producer flow control would kick in.
>>>> 
>>>> Hope this helps.
>>>> 
>>>> 
>>>> Torsten Mielke
>>>> torsten@fusesource.com
>>>> tmielke@blogspot.com
>>>> 
>>>> 
>>>> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> i have looked to the documentation of the producer flow control
>>>>> functionality (http://activemq.apache.org/producer-flow-control.html).
>>>>> 
>>>>> I see that it is based on the storage size but in my case, i think not
>>>>> being able to use the size.
>>>>> I have a queue where i put messages two kind of messages : Big 10MB and
>>>>> Small 1kB.
>>>>> 
>>>>> As Big messages are rare (compared to small ones), i would limit the
>>>>> Memory
>>>>> to 100MB (10 messages).
>>>>> But for small messages i would limit the number of small messages to
>> 5000
>>>>> (so about 5MB) . We have noted that with JDBC persistence, the number
>> of
>>>>> pending messages in the broker (not by queue) is important for
>>>>> "performances".
>>>>> 
>>>>> As i would keep the order of Big and Small messages, i put them in the
>>>>> same
>>>>> queue.
>>>>> 
>>>>> Is there a way to define a strategy for the flow control which is 100MB
>>>>> and
>>>>> 5k messages ?
>>>>> 
>>>>> 
>>>>> Regards
>>>>> Hervé


Mime
View raw message