activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: cursor memory usage/limit vs memory usage/limit
Date Thu, 01 Aug 2013 21:59:30 GMT
So your test case very clearly shows that the behavior is as I described
before.

I'm not sure why this is as it would make sense for durable subscriptions
to behave similarly to queues in regard to persistence and caching and how
it affects PFC.

I have a second test case that I've drawn up that might help me understand
a little better.

I'll let you know if I find something else.


On Thu, Aug 1, 2013 at 9:51 AM, Christian Posta
<christian.posta@gmail.com>wrote:

> actually looking through it right now :)
>
>
>
> On Thu, Aug 1, 2013 at 9:47 AM, Jake Choi <jw@yahoo-inc.com> wrote:
>
>> @Christian: Sorry for bugging you bug wonder if you've gotten a chance to
>> test the queue case as well, as I suggested.  Or please shed the light
>> once
>> more if I'm still missing or misunderstanding something.
>>
>> Regards,
>> Jake
>>
>>
>> Jake Choi wrote
>> > Thanks for your test.  But I still don't understand your point, meaning
>> > what you explained with prefetch-limit doesn't make sense to me.
>> >
>> > As I understand, prefetch limit has no relation with PFC as it's there
>> to
>> > throttle prefetching to consumers and possibly block message consumption
>> > from the consumer who has a lot messages left unacknowledged:
>> > http://activemq.apache.org/what-is-the-prefetch-limit-for.html
>> >
>> > I understand PFC has relation with the backlog size (cached pending list
>> > size) but I think it should do only for non-persistent message
>> publishing.
>> > Why do we need to block persistent message publishing by memory limit?
>> > Persistent messages could be contributing to the memory usage but until
>> > the cache limit is reached with the high-watermark (70%).  After that,
>> > broker can disable the cache/stopping caching(cursor dispatching) the
>> > persistent message but keep writing only to the backed persistent store
>> > w/o increasing the cache size (memory usage) anymore.  This is not a new
>> > but the way how Queue works for persistent message publishing for now,
>> as
>> > you documented:
>> /
>> >> For queues, it also affects the store cursor’s high water mark. A queue
>> >> has different cursors for persistent and non-persistent messages. If we
>> >> hit the high water mark (a threshold of the destination’s memory
>> limit),
>> >> no more messages be cached ready to be dispatched, and non-persistent
>> >> messages can be purged to temp disk as necessary (if the StoreCursor
>> will
>> >> use FilePendingMessageCursor… otherwise it will just use a
>> >> VMPendingMessageCursor and won’t purge to temporary store).
>> /
>> >
>> > Could you please kindly do the same test with the Queue? (please find
>> > "testPersistentQueue" method) e.g. prefetchLimit=0 and publish
>> persistent
>> > messages to the Queue which is configured with Store-based cursor for
>> the
>> > back-ed storage.  Regardless of the prefetchLimit setting on the queue
>> > consumer side, you'll be able to publish as many as messages to the
>> > back-ed storage not being blocked by any memory limit until it hits the
>> > storage limit..
>> > ceposta wrote
>> >> Thanks for the test...
>> >> I played around with it for a bit, and everything seems to work as I
>> >> expected.
>> >>
>> >> Memory usage for durable subs on topics only kicks in when dispatch to
>> >> the
>> >> consumer has already hit the prefetch limit. That means, try altering
>> >> your
>> >> prefetch limits, and you'll notice that the number of messages that can
>> >> be
>> >> dispatched to a live durable consumer without hitting PFC changes along
>> >> with the consumer prefetch. Once you've hit the prefetch limit, all of
>> >> those messages get cached and will count against memory usage and
>> >> eventually you'll hit PFC.
>> >>
>> >> Does that make sense?
>> >>
>> >>
>> >> On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi &lt;
>>
>> >> jw@
>>
>> >> &gt; wrote:
>> >>
>> >>> I'm testing only durable subscribers (as non-durable subscribers are
>> >>> working
>> >>> as expected as all of their messages seem being treated as
>> >>> non-persistent).
>> >>> And, I purposely set the prefetch limit to 0 (not using prefetching
at
>> >>> all)
>> >>> to quickly generate the cache full of the TopicStorePrefetch class.
>> >>>
>> >>> Please check out the test code I'm using: "prefetchLimitForAll" member
>> >>> variable is there to adjust prefetch limits of all types of
>> >>> subscriptions.
>> >>>
>> >>> MemoryLimitTest.java
>> >>> &lt;
>> http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java&gt
>> ;
>> >>>
>> >>>
>> >>> ceposta wrote
>> >>> > Give me one more piece of information.
>> >>> >
>> >>> > What is the prefetch settings you're using for your topics (durable
>> >>> and
>> >>> > non-durable subscribers), and how many durable subscribers are
>> >>> connected?
>> >>> >
>> >>> >
>> >>> > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi &lt;
>> >>>
>> >>> > jw@
>> >>>
>> >>> > &gt; wrote:
>> >>> >
>> >>> >> Thanks a lot Christian for the well-explained documentation
and
>> this
>> >>> >> explains
>> >>> >> why I'm seeing the trouble with topic durable subscriptions
>> regarding
>> >>> the
>> >>> >> per-destination memory usage & limit.
>> >>> >>
>> >>> >> First, let me quote some from your writing:
>> >>> >>
>> >>> >> "Main Broker Memory, Destination Memory, Subscription Memory
>> >>> >>
>> >>> >> ...
>> >>> >>
>> >>> >> A destination, when it’s created, will create its own SystemUsage
>> >>> object
>> >>> >> (which creates its own separate Memory, Store, and Temp Usage
>> >>> objects)
>> >>> >> but
>> >>> >> it will set its parent to the be broker’s main SystemUsage
object.
>> A
>> >>> >> destination can have its memory limits tuned individually (but
not
>> >>> Store
>> >>> >> and
>> >>> >> Temp, those will still delegate to the parent). To set a
>> >>> destination’s
>> >>> >> memory limit:
>> >>> >> ...
>> >>> >>
>> >>> >> So the destination usage objects can be used to more finely
control
>> >>> >> MemoryUsage, but it will always coordinate with the Main memory
for
>> >>> all
>> >>> >> usage counts. This functionality can be used to limit the number
of
>> >>> >> messages
>> >>> >> that a destination keeps around so that a single destination
cannot
>> >>> >> starve
>> >>> >> other destinations. *For queues, it also affects the store
cursor’s
>> >>> high
>> >>> >> water mark. A queue has different cursors for persistent and
>> >>> >> non-persistent
>> >>> >> messages. If we hit the high water mark (a threshold of the
>> >>> destination’s
>> >>> >> memory limit), no more messages be cached ready to be dispatched,
>> and
>> >>> >> non-persistent messages can be purged to temp disk as necessary
(if
>> >>> the
>> >>> >> StoreCursor will use FilePendingMessageCursor… otherwise
it will
>> just
>> >>> use
>> >>> >> a
>> >>> >> VMPendingMessageCursor and won’t purge to temporary store).*
>> >>> >>
>> >>> >> If you don’t specify a memory limit for individual destinations,
>> the
>> >>> >> destination’s SystemUsage will delegate to the parent (Main
>> >>> SystemUsage)
>> >>> >> for
>> >>> >> all usage counts. This means it will effectively use the broker’s
>> >>> Main
>> >>> >> SystemUsage for all memory-related counts.
>> >>> >>
>> >>> >> *Consumer subscriptions, on the other hand, don’t have any
notion
>> of
>> >>> >> their
>> >>> >> own SystemUsage or MemoryUsage counters. They will always use
the
>> >>> >> broker’s
>> >>> >> Main SystemUsage objects.* The main thing to note about this
is
>> when
>> >>> >> using
>> >>> >> a
>> >>> >> FilePendingMessageCursor for subscriptions (for example, for
a
>> Topic
>> >>> >> subscription), the messages will not be swapped to disk until
the
>> >>> cursor
>> >>> >> high-water mark (70% by default) is reached.. but that means
70% of
>> >>> Main
>> >>> >> memory will need to be reached. That could be a while, and
a lot of
>> >>> >> messages
>> >>> >> could be kept in memory. And if your subscription is the one
>> holding
>> >>> most
>> >>> >> of
>> >>> >> those messages, swapping to disk could take a while. As topics
>> >>> dispatch
>> >>> >> messages to one subscription at a time, if one subscription
grinds
>> to
>> >>> a
>> >>> >> halt
>> >>> >> because it’s swapping its messages to disk, the rest of the
>> >>> subscription
>> >>> >> ready to receive the message will also feel the slow down..."
>> >>> >>
>> >>> >> What I don't understand from the above is why consumer
>> subscriptions
>> >>> (and
>> >>> >> their cursors) of topics are not using per-destination MemoryLimit
>> >>> but
>> >>> >> share
>> >>> >> the Broker's main MemoryLimit, unlike the queues (please see
the
>> >>> bolded
>> >>> >> sentences above).  Due to this + some logics inside
>> >>> AbstractStoreCursor's
>> >>> >> space checking logics, PFC always kicks in for the topic whenever
>> the
>> >>> >> persistent store cursor's cache (pendingList) gets full.  Let
me
>> >>> explain
>> >>> >> what's happening:
>> >>> >>
>> >>> >> 1. A topic is created, with per-destination memory limit =
1MB and
>> >>> >> broker's
>> >>> >> main memory limit = 5MB.
>> >>> >> 2. Topic's SystemUsage (for PFC purpose) is configured with
>> >>> >> per-destination
>> >>> >> memory limit (1MB), while TopicStorePrefetch(persistent store
>> >>> cursor)'s
>> >>> >> SystemUsage (for remaining cache space checking purpose) is
>> >>> configured
>> >>> >> with
>> >>> >> broker's main memory limit (10MB).
>> >>> >> 3. The 1st message of 0.8MB is published:
>> >>> >>    a. PFC doesn't kick in at Topic layer, as
>> Topic#memoryUsage#isFull
>> >>> is
>> >>> >> true.
>> >>> >>    b. It's cached to the persistent cursor, as
>> >>> >> AbstractStoreCursor#hasSpace
>> >>> >> is true.
>> >>> >>    c. per-dest memory usage % becomes 90%, while main memory
usage
>> %
>> >>> >> becomes
>> >>> >> 9%.
>> >>> >> 4. The 2nd message of 0.8MB is published:
>> >>> >>    a. PFC doesn't kick in either, as per-dest memory usage
is 90%
>> (<
>> >>> >> 100%).
>> >>> >> This is fine.
>> >>> >>    b. *(I expect here that this second message shouldn't be
cached
>> to
>> >>> the
>> >>> >> cursor but invalidate it, but)this second message is also cached
to
>> >>> the
>> >>> >> persistent cursor, as AbstractStoreCursor#hasSpace is still
true:
>> 9%
>> >>> <
>> >>> >> 70%
>> >>> >> (main memory usage high watermark)!!!*
>> >>> >>    c. per-dest memory usage % becomes 180%, while main memory
>> usage %
>> >>> >> becomes 18%.
>> >>> >> 5. The 3rd message of 0.8MB is published:
>> >>> >>    a. *(I don't want this behavior but) PFC kicks in!!!* because
>> >>> >> per-destination memory limit is exceeded (180%).
>> >>> >>
>> >>> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch
checks
>> >>> >> cursor
>> >>> >> memory availability against per-dest memory limit (not against
>> >>> broker's
>> >>> >> main
>> >>> >> memory limit) so that cursor pending cache gets disabled (just
>> keeps
>> >>> what
>> >>> >> has been cached) w/o increasing the per-destination memory
usage.
>> >>> >>
>> >>> >> I don't want to block persistent message publishing to topics
by
>> the
>> >>> >> per-destination memory limit but only for non-persistent message
>> >>> >> publishing,
>> >>> >> cause the store size allowed for a destination is relatively
higher
>> >>> than
>> >>> >> the
>> >>> >> memory limit: e.g. I'd like to allow per-destination persistent
>> >>> message
>> >>> >> publishing up to 10GB disk space without being blocked by the
>> memory
>> >>> >> limit
>> >>> >> &
>> >>> >> PFC...  How can I achieve this?
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> View this message in context:
>> >>> >>
>> >>>
>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html
>> >>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > *Christian Posta*
>> >>> > http://www.christianposta.com/blog
>> >>> > twitter: @christianposta
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html
>> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> *Christian Posta*
>> >> http://www.christianposta.com/blog
>> >> twitter: @christianposta
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4670001.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message