activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Choi ...@yahoo-inc.com>
Subject Re: cursor memory usage/limit vs memory usage/limit
Date Thu, 01 Aug 2013 16:47:52 GMT
@Christian: Sorry for bugging you bug wonder if you've gotten a chance to
test the queue case as well, as I suggested.  Or please shed the light once
more if I'm still missing or misunderstanding something.

Regards,
Jake


Jake Choi wrote
> Thanks for your test.  But I still don't understand your point, meaning
> what you explained with prefetch-limit doesn't make sense to me.
> 
> As I understand, prefetch limit has no relation with PFC as it's there to
> throttle prefetching to consumers and possibly block message consumption
> from the consumer who has a lot messages left unacknowledged:
> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
> 
> I understand PFC has relation with the backlog size (cached pending list
> size) but I think it should do only for non-persistent message publishing. 
> Why do we need to block persistent message publishing by memory limit?
> Persistent messages could be contributing to the memory usage but until
> the cache limit is reached with the high-watermark (70%).  After that,
> broker can disable the cache/stopping caching(cursor dispatching) the
> persistent message but keep writing only to the backed persistent store
> w/o increasing the cache size (memory usage) anymore.  This is not a new
> but the way how Queue works for persistent message publishing for now, as
> you documented:
/
>> For queues, it also affects the store cursor’s high water mark. A queue
>> has different cursors for persistent and non-persistent messages. If we
>> hit the high water mark (a threshold of the destination’s memory limit),
>> no more messages be cached ready to be dispatched, and non-persistent
>> messages can be purged to temp disk as necessary (if the StoreCursor will
>> use FilePendingMessageCursor… otherwise it will just use a
>> VMPendingMessageCursor and won’t purge to temporary store).
/
> 
> Could you please kindly do the same test with the Queue? (please find
> "testPersistentQueue" method) e.g. prefetchLimit=0 and publish persistent
> messages to the Queue which is configured with Store-based cursor for the
> back-ed storage.  Regardless of the prefetchLimit setting on the queue
> consumer side, you'll be able to publish as many as messages to the
> back-ed storage not being blocked by any memory limit until it hits the
> storage limit..
> ceposta wrote
>> Thanks for the test...
>> I played around with it for a bit, and everything seems to work as I
>> expected.
>> 
>> Memory usage for durable subs on topics only kicks in when dispatch to
>> the
>> consumer has already hit the prefetch limit. That means, try altering
>> your
>> prefetch limits, and you'll notice that the number of messages that can
>> be
>> dispatched to a live durable consumer without hitting PFC changes along
>> with the consumer prefetch. Once you've hit the prefetch limit, all of
>> those messages get cached and will count against memory usage and
>> eventually you'll hit PFC.
>> 
>> Does that make sense?
>> 
>> 
>> On Mon, Jul 29, 2013 at 9:16 AM, Jake Choi <

>> jw@

>> > wrote:
>> 
>>> I'm testing only durable subscribers (as non-durable subscribers are
>>> working
>>> as expected as all of their messages seem being treated as
>>> non-persistent).
>>> And, I purposely set the prefetch limit to 0 (not using prefetching at
>>> all)
>>> to quickly generate the cache full of the TopicStorePrefetch class.
>>>
>>> Please check out the test code I'm using: "prefetchLimitForAll" member
>>> variable is there to adjust prefetch limits of all types of
>>> subscriptions.
>>>
>>> MemoryLimitTest.java
>>> <http://activemq.2283324.n4.nabble.com/file/n4669838/MemoryLimitTest.java>
>>>
>>>
>>> ceposta wrote
>>> > Give me one more piece of information.
>>> >
>>> > What is the prefetch settings you're using for your topics (durable
>>> and
>>> > non-durable subscribers), and how many durable subscribers are
>>> connected?
>>> >
>>> >
>>> > On Sun, Jul 28, 2013 at 8:40 PM, Jake Choi <
>>>
>>> > jw@
>>>
>>> > > wrote:
>>> >
>>> >> Thanks a lot Christian for the well-explained documentation and this
>>> >> explains
>>> >> why I'm seeing the trouble with topic durable subscriptions regarding
>>> the
>>> >> per-destination memory usage & limit.
>>> >>
>>> >> First, let me quote some from your writing:
>>> >>
>>> >> "Main Broker Memory, Destination Memory, Subscription Memory
>>> >>
>>> >> ...
>>> >>
>>> >> A destination, when it’s created, will create its own SystemUsage
>>> object
>>> >> (which creates its own separate Memory, Store, and Temp Usage
>>> objects)
>>> >> but
>>> >> it will set its parent to the be broker’s main SystemUsage object.
A
>>> >> destination can have its memory limits tuned individually (but not
>>> Store
>>> >> and
>>> >> Temp, those will still delegate to the parent). To set a
>>> destination’s
>>> >> memory limit:
>>> >> ...
>>> >>
>>> >> So the destination usage objects can be used to more finely control
>>> >> MemoryUsage, but it will always coordinate with the Main memory for
>>> all
>>> >> usage counts. This functionality can be used to limit the number of
>>> >> messages
>>> >> that a destination keeps around so that a single destination cannot
>>> >> starve
>>> >> other destinations. *For queues, it also affects the store cursor’s
>>> high
>>> >> water mark. A queue has different cursors for persistent and
>>> >> non-persistent
>>> >> messages. If we hit the high water mark (a threshold of the
>>> destination’s
>>> >> memory limit), no more messages be cached ready to be dispatched, and
>>> >> non-persistent messages can be purged to temp disk as necessary (if
>>> the
>>> >> StoreCursor will use FilePendingMessageCursor… otherwise it will just
>>> use
>>> >> a
>>> >> VMPendingMessageCursor and won’t purge to temporary store).*
>>> >>
>>> >> If you don’t specify a memory limit for individual destinations, the
>>> >> destination’s SystemUsage will delegate to the parent (Main
>>> SystemUsage)
>>> >> for
>>> >> all usage counts. This means it will effectively use the broker’s
>>> Main
>>> >> SystemUsage for all memory-related counts.
>>> >>
>>> >> *Consumer subscriptions, on the other hand, don’t have any notion
of
>>> >> their
>>> >> own SystemUsage or MemoryUsage counters. They will always use the
>>> >> broker’s
>>> >> Main SystemUsage objects.* The main thing to note about this is when
>>> >> using
>>> >> a
>>> >> FilePendingMessageCursor for subscriptions (for example, for a Topic
>>> >> subscription), the messages will not be swapped to disk until the
>>> cursor
>>> >> high-water mark (70% by default) is reached.. but that means 70% of
>>> Main
>>> >> memory will need to be reached. That could be a while, and a lot of
>>> >> messages
>>> >> could be kept in memory. And if your subscription is the one holding
>>> most
>>> >> of
>>> >> those messages, swapping to disk could take a while. As topics
>>> dispatch
>>> >> messages to one subscription at a time, if one subscription grinds to
>>> a
>>> >> halt
>>> >> because it’s swapping its messages to disk, the rest of the
>>> subscription
>>> >> ready to receive the message will also feel the slow down..."
>>> >>
>>> >> What I don't understand from the above is why consumer subscriptions
>>> (and
>>> >> their cursors) of topics are not using per-destination MemoryLimit
>>> but
>>> >> share
>>> >> the Broker's main MemoryLimit, unlike the queues (please see the
>>> bolded
>>> >> sentences above).  Due to this + some logics inside
>>> AbstractStoreCursor's
>>> >> space checking logics, PFC always kicks in for the topic whenever the
>>> >> persistent store cursor's cache (pendingList) gets full.  Let me
>>> explain
>>> >> what's happening:
>>> >>
>>> >> 1. A topic is created, with per-destination memory limit = 1MB and
>>> >> broker's
>>> >> main memory limit = 5MB.
>>> >> 2. Topic's SystemUsage (for PFC purpose) is configured with
>>> >> per-destination
>>> >> memory limit (1MB), while TopicStorePrefetch(persistent store
>>> cursor)'s
>>> >> SystemUsage (for remaining cache space checking purpose) is
>>> configured
>>> >> with
>>> >> broker's main memory limit (10MB).
>>> >> 3. The 1st message of 0.8MB is published:
>>> >>    a. PFC doesn't kick in at Topic layer, as Topic#memoryUsage#isFull
>>> is
>>> >> true.
>>> >>    b. It's cached to the persistent cursor, as
>>> >> AbstractStoreCursor#hasSpace
>>> >> is true.
>>> >>    c. per-dest memory usage % becomes 90%, while main memory usage %
>>> >> becomes
>>> >> 9%.
>>> >> 4. The 2nd message of 0.8MB is published:
>>> >>    a. PFC doesn't kick in either, as per-dest memory usage is 90% (<
>>> >> 100%).
>>> >> This is fine.
>>> >>    b. *(I expect here that this second message shouldn't be cached to
>>> the
>>> >> cursor but invalidate it, but)this second message is also cached to
>>> the
>>> >> persistent cursor, as AbstractStoreCursor#hasSpace is still true: 9%
>>> <
>>> >> 70%
>>> >> (main memory usage high watermark)!!!*
>>> >>    c. per-dest memory usage % becomes 180%, while main memory usage
%
>>> >> becomes 18%.
>>> >> 5. The 3rd message of 0.8MB is published:
>>> >>    a. *(I don't want this behavior but) PFC kicks in!!!* because
>>> >> per-destination memory limit is exceeded (180%).
>>> >>
>>> >> The behavior I wanted to see is: at #4.b. TopicStorePrefetch checks
>>> >> cursor
>>> >> memory availability against per-dest memory limit (not against
>>> broker's
>>> >> main
>>> >> memory limit) so that cursor pending cache gets disabled (just keeps
>>> what
>>> >> has been cached) w/o increasing the per-destination memory usage.
>>> >>
>>> >> I don't want to block persistent message publishing to topics by the
>>> >> per-destination memory limit but only for non-persistent message
>>> >> publishing,
>>> >> cause the store size allowed for a destination is relatively higher
>>> than
>>> >> the
>>> >> memory limit: e.g. I'd like to allow per-destination persistent
>>> message
>>> >> publishing up to 10GB disk space without being blocked by the memory
>>> >> limit
>>> >> &
>>> >> PFC...  How can I achieve this?
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669819.html
>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > *Christian Posta*
>>> > http://www.christianposta.com/blog
>>> > twitter: @christianposta
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4669838.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>> 
>> 
>> 
>> -- 
>> *Christian Posta*
>> http://www.christianposta.com/blog
>> twitter: @christianposta





--
View this message in context: http://activemq.2283324.n4.nabble.com/cursor-memory-usage-limit-vs-memory-usage-limit-tp4669679p4670001.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message