activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hervé BARRAULT <herve.barra...@gmail.com>
Subject Re: Producer flow control
Date Wed, 15 Feb 2012 14:48:28 GMT
Hi,
If i set cursorMemoryHighWaterMark to 100, is it a good idea to use
vmCursor instead of File Based Cursor (the default one) ?

Regards
Hervé

On Wed, Feb 15, 2012 at 10:50 AM, Torsten Mielke <torsten@fusesource.com>wrote:

> Hello,
>
> > So when using persistence, i can't rely on producerFlowControl to
> > limit the Flow.
>
>
> Yes you can. However when using persistent queue messages,
> producerFlowControl does not kick-in that early. Which is actually a good
> thing in general.
>
> All this is configurable as well. You can specify
> cursorMemoryHighWaterMark="100" on your destination policy configuration.
> With that the cursor will take up to 100% of the configured queue memory
> limit. When that limit is reached, producer flow control will also kick in
> (as now 100% of the memory are in use).
>
>
> > What is the best way to add a mechanism to constrain the queue size
> > (handling slow consumer when using persistence) ?
>
> ProducerFlowControl and perhaps in your case also setting
> cursorMemoryHighWaterMark="100" on the destination policy.
>
>
> Best Regards,
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>
>
> On Feb 14, 2012, at 2:20 PM, Hervé BARRAULT wrote:
>
> > Hi,
> > thanks for the answer.
> >
> > So when using persistence, i can't rely on producerFlowControl to
> > limit the Flow.
> >
> > If i have a consumer which is able to process only 10 messages by
> > seconds and my queue is filled with 36000 messages (due to faster
> > producers), i know that it will take 1 hour to process all my
> > messages. I would reject new messages as my processing time shall be
> > less than one hour.
> >
> > I would also keep the order of incoming messages (small and big) so i
> > can't easily manage two queues without adding a complex
> > synchronization mechanism.
> >
> > I see http://activemq.apache.org/slow-consumer-handling.html for
> > non-persistent messages but it should not apply for persistent ones.
> > (Prefetch is configurable for persistence :
> >
> http://activemq.apache.org/slow-consumer-handling.html#SlowConsumerHandling-UsingthePrefetchPolicytoconfigurethelimit
> > but nothing for Pending Message Limit Strategy).
> >
> > What is the best way to add a mechanism to constrain the queue size
> > (handling slow consumer when using persistence) ?
> >
> > Modifying the cursor, producer flow control, another thing ?
> >
> > Regards
> >
> > Hervé
> >
> >
> > On 2/14/12, Torsten Mielke <torsten@fusesource.com> wrote:
> >> Hello Herve,
> >>
> >> You cannot configure producer flow control based on the various message
> >> sizes.
> >> However you could consider using different queues for these types of
> msgs
> >> and configure
> >> the limits for each queue differently. That way your small msgs won't be
> >> blocked just because of a small amount of large msgs on the same queue
> >> (assuming different producers for both kinds of msgs).
> >>
> >> Also, when sending persistent queue messages, you rarely see producer
> flow
> >> control happening when using the default store cursor. That is because
> the
> >> store cursor is configured to hold messages in memory up to only 70% of
> the
> >> configured destination limit. If this 70% limit is reached, it won't
> keep
> >> new msgs in memory but reload them from the store (kahadb) later (when
> its
> >> time to dispatch the msg). Remember a persistent msg is first written
> to the
> >> store when received by the broker.
> >> Producer flow control however kicks in when 100% of the available
> memory for
> >> each destination are used. This limit however you generally don't reach.
> >> Things are different for non-persistent msgs. The default file cursor
> keeps
> >> msgs in memory until 100% of the configured destination limit are
> reached.
> >> Thereafter it swaps msgs to temp storage (which is different from
> kahadb)
> >> and producer flow control would kick in.
> >>
> >> Hope this helps.
> >>
> >>
> >> Torsten Mielke
> >> torsten@fusesource.com
> >> tmielke@blogspot.com
> >>
> >>
> >> On Feb 13, 2012, at 2:37 PM, Hervé BARRAULT wrote:
> >>
> >>> Hi,
> >>>
> >>> i have looked to the documentation of the producer flow control
> >>> functionality (http://activemq.apache.org/producer-flow-control.html).
> >>>
> >>> I see that it is based on the storage size but in my case, i think not
> >>> being able to use the size.
> >>> I have a queue where i put messages two kind of messages : Big 10MB and
> >>> Small 1kB.
> >>>
> >>> As Big messages are rare (compared to small ones), i would limit the
> >>> Memory
> >>> to 100MB (10 messages).
> >>> But for small messages i would limit the number of small messages to
> 5000
> >>> (so about 5MB) . We have noted that with JDBC persistence, the number
> of
> >>> pending messages in the broker (not by queue) is important for
> >>> "performances".
> >>>
> >>> As i would keep the order of Big and Small messages, i put them in the
> >>> same
> >>> queue.
> >>>
> >>> Is there a way to define a strategy for the flow control which is 100MB
> >>> and
> >>> 5k messages ?
> >>>
> >>>
> >>> Regards
> >>> Hervé
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message