activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Choi ...@yahoo-inc.com>
Subject Re: cursor memory usage/limit vs memory usage/limit
Date Mon, 29 Jul 2013 16:16:52 GMT
I'm testing only durable subscribers (as non-durable subscribers are working
as expected as all of their messages seem being treated as non-persistent). 
And, I purposely set the prefetch limit to 0 (not using prefetching at all)
to quickly generate the cache full of the TopicStorePrefetch class.

Please check out the test code I'm using: "prefetchLimitForAll" member
variable is there to adjust prefetch limits of all types of subscriptions.

MemoryLimitTest.java
<http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java>  


ceposta wrote
> Give me one more piece of information.
> 
> What is the prefetch settings you're using for your topics (durable and
> non-durable subscribers), and how many durable subscribers are connected?
> 
> 
> On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi &lt;

> jw@

> &gt; wrote:
> 
>> Thanks a lot Christian for the well-explained documentation and this
>> explains
>> why I'm seeing the trouble with topic durable subscriptions regarding the
>> per-destination memory usage & limit.
>>
>> First, let me quote some from your writing:
>>
>> "Main Broker Memory, Destination Memory, Subscription Memory
>>
>> ...
>>
>> A destination, when it’s created, will create its own SystemUsage object
>> (which creates its own separate Memory, Store, and Temp Usage objects)
>> but
>> it will set its parent to the be broker’s main SystemUsage object. A
>> destination can have its memory limits tuned individually (but not Store
>> and
>> Temp, those will still delegate to the parent). To set a destination’s
>> memory limit:
>> ...
>>
>> So the destination usage objects can be used to more finely control
>> MemoryUsage, but it will always coordinate with the Main memory for all
>> usage counts. This functionality can be used to limit the number of
>> messages
>> that a destination keeps around so that a single destination cannot
>> starve
>> other destinations. *For queues, it also affects the store cursor’s high
>> water mark. A queue has different cursors for persistent and
>> non-persistent
>> messages. If we hit the high water mark (a threshold of the destination’s
>> memory limit), no more messages be cached ready to be dispatched, and
>> non-persistent messages can be purged to temp disk as necessary (if the
>> StoreCursor will use FilePendingMessageCursor… otherwise it will just use
>> a
>> VMPendingMessageCursor and won’t purge to temporary store).*
>>
>> If you don’t specify a memory limit for individual destinations, the
>> destination’s SystemUsage will delegate to the parent (Main SystemUsage)
>> for
>> all usage counts. This means it will effectively use the broker’s Main
>> SystemUsage for all memory-related counts.
>>
>> *Consumer subscriptions, on the other hand, don’t have any notion of
>> their
>> own SystemUsage or MemoryUsage counters. They will always use the
>> broker’s
>> Main SystemUsage objects.* The main thing to note about this is when
>> using
>> a
>> FilePendingMessageCursor for subscriptions (for example, for a Topic
>> subscription), the messages will not be swapped to disk until the cursor
>> high-water mark (70% by default) is reached.. but that means 70% of Main
>> memory will need to be reached. That could be a while, and a lot of
>> messages
>> could be kept in memory. And if your subscription is the one holding most
>> of
>> those messages, swapping to disk could take a while. As topics dispatch
>> messages to one subscription at a time, if one subscription grinds to a
>> halt
>> because it’s swapping its messages to disk, the rest of the subscription
>> ready to receive the message will also feel the slow down..."
>>
>> What I don't understand from the above is why consumer subscriptions (and
>> their cursors) of topics are not using per-destination MemoryLimit but
>> share
>> the Broker's main MemoryLimit, unlike the queues (please see the bolded
>> sentences above).  Due to this + some logics inside AbstractStoreCursor's
>> space checking logics, PFC always kicks in for the topic whenever the
>> persistent store cursor's cache (pendingList) gets full.  Let me explain
>> what's happening:
>>
>> 1. A topic is created, with per-destination memory limit = 1MB and
>> broker's
>> main memory limit = 5MB.
>> 2. Topic's SystemUsage (for PFC purpose) is configured with
>> per-destination
>> memory limit (1MB), while TopicStorePrefetch(persistent store cursor)'s
>> SystemUsage (for remaining cache space checking purpose) is configured
>> with
>> broker's main memory limit (10MB).
>> 3. The 1st message of 0.8MB is published:
>>    a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull is
>> true.
>>    b. It's cached to the persistent cursor, as
>> AbstractStoreCursor#hasSpace
>> is true.
>>    c. per-dest memory usage % becomes 90%, while main memory usage %
>> becomes
>> 9%.
>> 4. The 2nd message of 0.8MB is published:
>>    a. PFC doesn't kick in either, as per-dest memory usage is 90% (<
>> 100%).
>> This is fine.
>>    b. *(I expect here that this second message shouldn't be cached to the
>> cursor but invalidate it, but)this second message is also cached to the
>> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9% <
>> 70%
>> (main memory usage high watermark)!!!*
>>    c. per-dest memory usage % becomes 180%, while main memory usage %
>> becomes 18%.
>> 5. The 3rd message of 0.8MB is published:
>>    a. *(I don't want this behavior but) PFC kicks in!!!* because
>> per-destination memory limit is exceeded (180%).
>>
>> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks
>> cursor
>> memory availability against per-dest memory limit (not against broker's
>> main
>> memory limit) so that cursor pending cache gets disabled (just keeps what
>> has been cached) w/o increasing the per-destination memory usage.
>>
>> I don't want to block persistent message publishing to topics by the
>> per-destination memory limit but only for non-persistent message
>> publishing,
>> cause the store size allowed for a destination is relatively higher than
>> the
>> memory limit: e.g. I'd like to allow per-destination persistent message
>> publishing up to 10GB disk space without being blocked by the memory
>> limit
>> &
>> PFC...  How can I achieve this?
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 
> -- 
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta





--
View this message in context: http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message