activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: cursor memory usage/limit vs memory usage/limit
Date Thu, 01 Aug 2013 16:51:22 GMT
actually looking through it right now :)



On Thu, Aug 1, 2013 at 9:47 AM, Jake Choi <jw@yahoo-inc.com> wrote:

> @Christian: Sorry for bugging you bug wonder if you've gotten a chance to
> test the queue case as well, as I suggested.  Or please shed the light once
> more if I'm still missing or misunderstanding something.
>
> Regards,
> Jake
>
>
> Jake Choi wrote
> > Thanks for your test.  But I still don't understand your point, meaning
> > what you explained with prefetch-limit doesn't make sense to me.
> >
> > As I understand, prefetch limit has no relation with PFC as it's there to
> > throttle prefetching to consumers and possibly block message consumption
> > from the consumer who has a lot messages left unacknowledged:
> > http://activemq.apache.org/what-is-the-prefetch-limit-for.html
> >
> > I understand PFC has relation with the backlog size (cached pending list
> > size) but I think it should do only for non-persistent message
> publishing.
> > Why do we need to block persistent message publishing by memory limit?
> > Persistent messages could be contributing to the memory usage but until
> > the cache limit is reached with the high-watermark (70%).  After that,
> > broker can disable the cache/stopping caching(cursor dispatching) the
> > persistent message but keep writing only to the backed persistent store
> > w/o increasing the cache size (memory usage) anymore.  This is not a new
> > but the way how Queue works for persistent message publishing for now, as
> > you documented:
> /
> >> For queues, it also affects the store cursor’s high water mark. A queue
> >> has different cursors for persistent and non-persistent messages. If we
> >> hit the high water mark (a threshold of the destination’s memory limit),
> >> no more messages be cached ready to be dispatched, and non-persistent
> >> messages can be purged to temp disk as necessary (if the StoreCursor
> will
> >> use FilePendingMessageCursor… otherwise it will just use a
> >> VMPendingMessageCursor and won’t purge to temporary store).
> /
> >
> > Could you please kindly do the same test with the Queue? (please find
> > "testPersistentQueue" method) e.g. prefetchLimit=0 and publish persistent
> > messages to the Queue which is configured with Store-based cursor for the
> > back-ed storage.  Regardless of the prefetchLimit setting on the queue
> > consumer side, you'll be able to publish as many as messages to the
> > back-ed storage not being blocked by any memory limit until it hits the
> > storage limit..
> > ceposta wrote
> >> Thanks for the test...
> >> I played around with it for a bit, and everything seems to work as I
> >> expected.
> >>
> >> Memory usage for durable subs on topics only kicks in when dispatch to
> >> the
> >> consumer has already hit the prefetch limit. That means, try altering
> >> your
> >> prefetch limits, and you'll notice that the number of messages that can
> >> be
> >> dispatched to a live durable consumer without hitting PFC changes along
> >> with the consumer prefetch. Once you've hit the prefetch limit, all of
> >> those messages get cached and will count against memory usage and
> >> eventually you'll hit PFC.
> >>
> >> Does that make sense?
> >>
> >>
> >> On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi &lt;
>
> >> jw@
>
> >> &gt; wrote:
> >>
> >>> I'm testing only durable subscribers (as non-durable subscribers are
> >>> working
> >>> as expected as all of their messages seem being treated as
> >>> non-persistent).
> >>> And, I purposely set the prefetch limit to 0 (not using prefetching at
> >>> all)
> >>> to quickly generate the cache full of the TopicStorePrefetch class.
> >>>
> >>> Please check out the test code I'm using: "prefetchLimitForAll" member
> >>> variable is there to adjust prefetch limits of all types of
> >>> subscriptions.
> >>>
> >>> MemoryLimitTest.java
> >>> &lt;
> http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java&gt
> ;
> >>>
> >>>
> >>> ceposta wrote
> >>> > Give me one more piece of information.
> >>> >
> >>> > What is the prefetch settings you're using for your topics (durable
> >>> and
> >>> > non-durable subscribers), and how many durable subscribers are
> >>> connected?
> >>> >
> >>> >
> >>> > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi &lt;
> >>>
> >>> > jw@
> >>>
> >>> > &gt; wrote:
> >>> >
> >>> >> Thanks a lot Christian for the well-explained documentation and
this
> >>> >> explains
> >>> >> why I'm seeing the trouble with topic durable subscriptions
> regarding
> >>> the
> >>> >> per-destination memory usage & limit.
> >>> >>
> >>> >> First, let me quote some from your writing:
> >>> >>
> >>> >> "Main Broker Memory, Destination Memory, Subscription Memory
> >>> >>
> >>> >> ...
> >>> >>
> >>> >> A destination, when it’s created, will create its own SystemUsage
> >>> object
> >>> >> (which creates its own separate Memory, Store, and Temp Usage
> >>> objects)
> >>> >> but
> >>> >> it will set its parent to the be broker’s main SystemUsage object.
A
> >>> >> destination can have its memory limits tuned individually (but
not
> >>> Store
> >>> >> and
> >>> >> Temp, those will still delegate to the parent). To set a
> >>> destination’s
> >>> >> memory limit:
> >>> >> ...
> >>> >>
> >>> >> So the destination usage objects can be used to more finely control
> >>> >> MemoryUsage, but it will always coordinate with the Main memory
for
> >>> all
> >>> >> usage counts. This functionality can be used to limit the number
of
> >>> >> messages
> >>> >> that a destination keeps around so that a single destination cannot
> >>> >> starve
> >>> >> other destinations. *For queues, it also affects the store cursor’s
> >>> high
> >>> >> water mark. A queue has different cursors for persistent and
> >>> >> non-persistent
> >>> >> messages. If we hit the high water mark (a threshold of the
> >>> destination’s
> >>> >> memory limit), no more messages be cached ready to be dispatched,
> and
> >>> >> non-persistent messages can be purged to temp disk as necessary
(if
> >>> the
> >>> >> StoreCursor will use FilePendingMessageCursor… otherwise it will
> just
> >>> use
> >>> >> a
> >>> >> VMPendingMessageCursor and won’t purge to temporary store).*
> >>> >>
> >>> >> If you don’t specify a memory limit for individual destinations,
the
> >>> >> destination’s SystemUsage will delegate to the parent (Main
> >>> SystemUsage)
> >>> >> for
> >>> >> all usage counts. This means it will effectively use the broker’s
> >>> Main
> >>> >> SystemUsage for all memory-related counts.
> >>> >>
> >>> >> *Consumer subscriptions, on the other hand, don’t have any notion
of
> >>> >> their
> >>> >> own SystemUsage or MemoryUsage counters. They will always use the
> >>> >> broker’s
> >>> >> Main SystemUsage objects.* The main thing to note about this is
when
> >>> >> using
> >>> >> a
> >>> >> FilePendingMessageCursor for subscriptions (for example, for a
Topic
> >>> >> subscription), the messages will not be swapped to disk until the
> >>> cursor
> >>> >> high-water mark (70% by default) is reached.. but that means 70%
of
> >>> Main
> >>> >> memory will need to be reached. That could be a while, and a lot
of
> >>> >> messages
> >>> >> could be kept in memory. And if your subscription is the one holding
> >>> most
> >>> >> of
> >>> >> those messages, swapping to disk could take a while. As topics
> >>> dispatch
> >>> >> messages to one subscription at a time, if one subscription grinds
> to
> >>> a
> >>> >> halt
> >>> >> because it’s swapping its messages to disk, the rest of the
> >>> subscription
> >>> >> ready to receive the message will also feel the slow down..."
> >>> >>
> >>> >> What I don't understand from the above is why consumer subscriptions
> >>> (and
> >>> >> their cursors) of topics are not using per-destination MemoryLimit
> >>> but
> >>> >> share
> >>> >> the Broker's main MemoryLimit, unlike the queues (please see the
> >>> bolded
> >>> >> sentences above).  Due to this + some logics inside
> >>> AbstractStoreCursor's
> >>> >> space checking logics, PFC always kicks in for the topic whenever
> the
> >>> >> persistent store cursor's cache (pendingList) gets full.  Let me
> >>> explain
> >>> >> what's happening:
> >>> >>
> >>> >> 1. A topic is created, with per-destination memory limit = 1MB
and
> >>> >> broker's
> >>> >> main memory limit = 5MB.
> >>> >> 2. Topic's SystemUsage (for PFC purpose) is configured with
> >>> >> per-destination
> >>> >> memory limit (1MB), while TopicStorePrefetch(persistent store
> >>> cursor)'s
> >>> >> SystemUsage (for remaining cache space checking purpose) is
> >>> configured
> >>> >> with
> >>> >> broker's main memory limit (10MB).
> >>> >> 3. The 1st message of 0.8MB is published:
> >>> >>    a. PFC doesn't kick in at Topic layer, as
> Topic#memoryUsage#isFull
> >>> is
> >>> >> true.
> >>> >>    b. It's cached to the persistent cursor, as
> >>> >> AbstractStoreCursor#hasSpace
> >>> >> is true.
> >>> >>    c. per-dest memory usage % becomes 90%, while main memory usage
%
> >>> >> becomes
> >>> >> 9%.
> >>> >> 4. The 2nd message of 0.8MB is published:
> >>> >>    a. PFC doesn't kick in either, as per-dest memory usage is 90%
(<
> >>> >> 100%).
> >>> >> This is fine.
> >>> >>    b. *(I expect here that this second message shouldn't be cached
> to
> >>> the
> >>> >> cursor but invalidate it, but)this second message is also cached
to
> >>> the
> >>> >> persistent cursor, as AbstractStoreCursor#hasSpace is still true:
9%
> >>> <
> >>> >> 70%
> >>> >> (main memory usage high watermark)!!!*
> >>> >>    c. per-dest memory usage % becomes 180%, while main memory usage
> %
> >>> >> becomes 18%.
> >>> >> 5. The 3rd message of 0.8MB is published:
> >>> >>    a. *(I don't want this behavior but) PFC kicks in!!!* because
> >>> >> per-destination memory limit is exceeded (180%).
> >>> >>
> >>> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks
> >>> >> cursor
> >>> >> memory availability against per-dest memory limit (not against
> >>> broker's
> >>> >> main
> >>> >> memory limit) so that cursor pending cache gets disabled (just
keeps
> >>> what
> >>> >> has been cached) w/o increasing the per-destination memory usage.
> >>> >>
> >>> >> I don't want to block persistent message publishing to topics by
the
> >>> >> per-destination memory limit but only for non-persistent message
> >>> >> publishing,
> >>> >> cause the store size allowed for a destination is relatively higher
> >>> than
> >>> >> the
> >>> >> memory limit: e.g. I'd like to allow per-destination persistent
> >>> message
> >>> >> publishing up to 10GB disk space without being blocked by the memory
> >>> >> limit
> >>> >> &
> >>> >> PFC...  How can I achieve this?
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> View this message in context:
> >>> >>
> >>>
> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html
> >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > *Christian Posta*
> >>> > http://www.christianposta.com/blog
> >>> > twitter: @christianposta
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html
> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>>
> >>
> >>
> >>
> >> --
> >> *Christian Posta*
> >> http://www.christianposta.com/blog
> >> twitter: @christianposta
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4670001.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message