activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Understanding memoryUsage (once again!)
Date Tue, 27 Nov 2012 23:40:33 GMT
See inline...


On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <juanin@gmail.com> wrote:

> Hi!
>
> Sorry for the delay in replying, buried on a project.
>
> As I mentioned before, I had tested this with 5.7.0 with the same
> behaviour.
> I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and on
> my case it doesn't matter if there are consumers or not, it always seems to
> make usage of the memory.
>
> Although I guess in theory that should not affect, did you use Stomp for
> your testing, or maybe you used Openwire?
> I'm using Stomp for my testing.
>
> Might be though that the broker's memory itself is not going beyond 70% of
> memoryUsage, but this is just per destination counters as you mentioned.
> In which case I guess the value shown as "Memory percent used" is a bit
> confusing... But haven't had much time to really test the possibility of
> exhausting the broker's memory.
>
No, i believe what you're seeing is correct. The broker's memory limit is
going beyond memoryUsage (way beyond). When a queue checks whether memory
is full, it will only do something interesting if producer flow control is
enabled. Otherwise, it will continue on. You are seeing that it will
continue to add messages until the Queue's memory limit (40MB) reaches the
70% mark. Since MemoryUsages are hierarchical, this means it will also
account for messages in the overall broker memory as well. For each queue,
you'll see that it will continue to hold 70% of 40MB of memory. What you
want in this case (if there are no consumers, or slow consumers) is to
raise your system usage memory limit OR lower your per-destination limits
OR lower your cursor highwatermark or a combination of all three.

http://activemq.apache.org/per-destination-policies.html

With PFC turned off, you're essentially telling the broker to take the
message no matter what. There is a point at which you will run out of
resources (memory, disk, etc). The trick is to find your use case and tune
for that.


>
> Will try to do some more testing soon...
>
> Thanks
>
>
> On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> <christian.posta@gmail.com>wrote:
>
> > Can you please try on 5.7?
> > I just tried a test, and if there are no consumers to the queue then the
> > memory usage will stay at 0%. The message will not be retained, ie, it
> will
> > be put into the store and kept there. If I add a consumer, and not try to
> > consume, the message will be kept around in memory up to the cursor high
> > watermark (70 by default).
> >
> > As I add more queues the same behavior as described above will happen.
> If I
> > attach consumers to the queues without consuming them (so no messages are
> > consumed), then messages are kept in the cursor up to the high-water
> > mark... note.. the high-water mark is relative to the
> Destination/Cursor's
> > MemoryUsage, not the global memory usage.
> >
> > If I continue adding queues, and with producer flow control set to
> false, I
> > too will see the *Global* memory usage go much higher than 100%. This is
> > not surprising though, because as I understand, these usage memory
> objects
> > are really just counters. They don't enforce anything. When coupled with
> > producer flow control, they can be used to determine when to enable PFC.
> If
> > PFC is false, it's up to the cursor to determine when to flush out to
> disk.
> > But each destination/cursor will have it's own system usage (with the
> > global as the parent).
> >
> > Hope this helps. Can you please try with 5.7 and give us a report back?
> > Thanks,
> > Christian
> >
> >
> >
> > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <juanin@gmail.com> wrote:
> >
> > > nope, adding a 3rd queue the 3rd one also gets this same value, so even
> > if
> > > it's the memory usage of the queue it's anyway going beyond..
> > >
> > >
> > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <juanin@gmail.com> wrote:
> > >
> > > > Might it be just a bug on how the MemoryPercentUsage is calculated?
> > > >
> > > > If I connect via JMX using console, I can see the MemoryPercentUsage
> as
> > > > 112 right now.
> > > > If I go to each of the 2 queues on them I see CursorMemoryUsage with
> > > value
> > > > 29360604, which would be 28mb each, summing a total of 56mb (just a
> bit
> > > > more than the specified memoryUsage of 50mb).
> > > >
> > > > Not sure I'm interpreting these values correctly though, first time I
> > > > access it via jconsole...
> > > >
> > > >
> > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <juanin@gmail.com> wrote:
> > > >
> > > >> On that config there's a 40mb memoryLimit per queue, but also tested
> > it
> > > >> without it with same results.
> > > >>
> > > >>
> > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <juanin@gmail.com>
wrote:
> > > >>
> > > >>> Hi Torsten!
> > > >>>
> > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the
same
> > > >>> results...
> > > >>> This is my 5.3.2 config:
> > > >>>
> > > >>> <beans
> > > >>>   xmlns="http://www.springframework.org/schema/beans"
> > > >>>   xmlns:amq="http://activemq.apache.org/schema/core"
> > > >>>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > >>>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> > > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > >>>   http://activemq.apache.org/schema/core
> > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > >>>
> > > >>>     <bean
> > > >>>
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > >>>         <property name="locations">
> > > >>>
> > > >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> > > >>>         </property>
> > > >>>     </bean>
> > > >>>
> > > >>>     <broker xmlns="http://activemq.apache.org/schema/core"
> > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > > >>>
> > > >>>         <destinationPolicy>
> > > >>>             <policyMap>
> > > >>>                 <policyEntries>
> > > >>>                     <policyEntry topic=">"
> producerFlowControl="true"
> > > >>> memoryLimit="5mb">
> > > >>>                         <pendingSubscriberPolicy>
> > > >>>                             <vmCursor />
> > > >>>                         </pendingSubscriberPolicy>
> > > >>>                     </policyEntry>
> > > >>>                     <policyEntry queue=">"
> > producerFlowControl="false"
> > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > >>>                 <deadLetterStrategy>
> > > >>>                             <individualDeadLetterStrategy
> > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > >>>                         </deadLetterStrategy>
> > > >>>                     </policyEntry>
> > > >>>                 </policyEntries>
> > > >>>             </policyMap>
> > > >>>         </destinationPolicy>
> > > >>>
> > > >>>         <managementContext>
> > > >>>             <managementContext connectorPort="2011"/>
> > > >>>         </managementContext>
> > > >>>
> > > >>>         <persistenceAdapter>
> > > >>>             <kahaDB directory="${activemq.base}/data/kahadb"
> > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > >>> indexCacheSize="1000"/>
> > > >>>         </persistenceAdapter>
> > > >>>
> > > >>>         <systemUsage>
> > > >>>             <systemUsage>
> > > >>>               <memoryUsage>
> > > >>>                     <memoryUsage limit="50 mb"/>
> > > >>>                  </memoryUsage>
> > > >>>                 <storeUsage>
> > > >>>                     <storeUsage limit="1 gb" name="foo"/>
> > > >>>                 </storeUsage>
> > > >>>                 <tempUsage>
> > > >>>                     <tempUsage limit="3 gb"/>
> > > >>>                 </tempUsage>
> > > >>>             </systemUsage>
> > > >>>         </systemUsage>
> > > >>>
> > > >>>         <transportConnectors>
> > > >>>             <transportConnector name="openwire" uri="tcp://
> > > 0.0.0.0:61616
> > > >>> "/>
> > > >>>     <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
> > > >>>         </transportConnectors>
> > > >>>
> > > >>>     </broker>
> > > >>>
> > > >>>     <import resource="jetty.xml"/>
> > > >>>
> > > >>> </beans>
> > > >>>
> > > >>>
> > > >>> Using just a simple PHP script with Stomp for feeding the queues
> > > >>> (running it twice with different queue name):
> > > >>>
> > > >>> <?php
> > > >>>
> > > >>> require_once("Stomp.php");
> > > >>>
> > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > >>> $amq->connect();
> > > >>>
> > > >>> for($i=1; $i <= 100000; $i++)
> > > >>> {
> > > >>> if($i%1000 == 0)
> > > >>> {
> > > >>>  echo "\nmsg #: $i";
> > > >>> }
> > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > >>> ,array('persistent' => 'true'));
> > > >>> }
> > > >>>
> > > >>> $amq->disconnect();
> > > >>>
> > > >>> ?>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > torsten@fusesource.com>wrote:
> > > >>>
> > > >>>> Hello,
> > > >>>>
> > > >>>> See in-line response.
> > > >>>>
> > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > >>>>
> > > >>>> > Hi!
> > > >>>> >
> > > >>>> > After some heavy digging about Producer Flow control
and the
> > > >>>> systemUsage
> > > >>>> > properties a couple of years ago, I thought I quite understood
> it.
> > > >>>> > But yesterday I found that one of my configs was not
behaving
> > > exactly
> > > >>>> as I
> > > >>>> > expected, so started doing some tests, and I see certain
> > behaviours
> > > >>>> which
> > > >>>> > don't seem to match what the docs and posts that I find
on the
> > list
> > > or
> > > >>>> > other forums say.
> > > >>>> >
> > > >>>> > "storeUsage" is perfectly clear, it's the max space that
> > persistent
> > > >>>> > messages can use to be stored in disk.
> > > >>>> > "tempUsage"" applies to file cursors on non-persistent
messages,
> > so
> > > >>>> as to
> > > >>>> > flush to disk if memory limits are reached (I don't care
much
> > about
> > > >>>> this
> > > >>>> > one anyway, I always use persistent messages).
> > > >>>>
> > > >>>> Correct.
> > > >>>>
> > > >>>> >
> > > >>>> > Now, according to most posts, memoryUsage would be the
maximum
> > > memory
> > > >>>> that
> > > >>>> > the broker would be available to use.
> > > >>>> > On this post:
> > > >>>> >
> > > >>>>
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > >>>> > says that "memoryUsage corresponds to the amount of memory
> that's
> > > >>>> > assigned to the in-memory store".
> > > >>>>
> > > >>>> Correct.
> > > >>>>
> > > >>>> >
> > > >>>> > For example, on my tests using the following config (only
> showing
> > > >>>> relevant
> > > >>>> > parts):
> > > >>>> >
> > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > >>>> optimizedDispatch="true">
> > > >>>> >    <deadLetterStrategy>
> > > >>>> >        <individualDeadLetterStrategy queuePrefix="DLQ."
> > > >>>> > useQueueForQueueMessages="true" />
> > > >>>> >    </deadLetterStrategy>
> > > >>>> > </policyEntry>
> > > >>>> >
> > > >>>> > <systemUsage>
> > > >>>> >    <systemUsage>
> > > >>>> >        <memoryUsage>
> > > >>>> >            <memoryUsage limit="100 mb"/>
> > > >>>> >        </memoryUsage>
> > > >>>> >        <storeUsage>
> > > >>>> >            <storeUsage limit="1 gb" name="foo"/>
> > > >>>> >        </storeUsage>
> > > >>>> >        <tempUsage>
> > > >>>> >            <tempUsage limit="3 gb"/>
> > > >>>> >        </tempUsage>
> > > >>>> >    </systemUsage>
> > > >>>> > </systemUsage>
> > > >>>> >
> > > >>>> > With that config I would expect the broker to use 100
mb of
> > maximum
> > > >>>> memory
> > > >>>> > among all queues. So it could maybe use 30mb in one queue
and
> 70mb
> > > in
> > > >>>> > second queue.
> > > >>>> >
> > > >>>> >
> > > >>>> > 1) What I'm seeing is that if I start feeding a queue
without
> > > >>>> consuming it,
> > > >>>> > the "Memory percent used" grows up to 70%, after that
it doesn't
> > > grow
> > > >>>> > anymore.
> > > >>>> > What is it doing exactly there? The first 70% is stored
in
> memory
> > > >>>> (apart
> > > >>>> > from disk since it's persistent), and all the rest that
> continues
> > > >>>> being fed
> > > >>>> > goes just to disk?
> > > >>>>
> > > >>>> This behavior is correct. For queues the default cursor is
store
> > > >>>> cursor. It keeps any newly arrived msgs in memory as long
as it
> does
> > > not
> > > >>>> reach the configured memory limit (either configured on the
queue
> > per
> > > >>>> destination or globally in memoryUsage settings).
> > > >>>> Once the cursor reaches 70% of the configured limit (in your
case
> of
> > > >>>> the memoryUsage limit since you don't specify a per-destination
> > > limit), it
> > > >>>> will not keep any more messages in memory.
> > > >>>> Instead it will reload these messages from the store when
its time
> > to
> > > >>>> dispatch them. The broker anyway persists any msgs it receives
> > before
> > > >>>> passing on to the cursor.
> > > >>>> This limit of 70% can be configured and raised to e..g 100%.
> > > >>>> This behavior is kind of an optimization. That way you run
less
> > often
> > > >>>> into producer-flow-control.
> > > >>>> As long as the persistence store is not running full, there
is no
> > need
> > > >>>> to block producers, since the cursor can also load the messages
> from
> > > the
> > > >>>> store and does not necessarily have to keep them in memory.
> > > >>>> If you configure the vmQueueCursor, then the behavior is
> different.
> > > >>>> This cursor will not be able to load msgs to the store but
needs
> to
> > > keep
> > > >>>> them all in memory. The vmQueueCursor used to be the default
> cursor
> > in
> > > >>>> older version of AMQ.
> > > >>>>
> > > >>>> Also note that topic msgs and non-persistent queue messages
are
> not
> > > >>>> handled by the store cursor. These msgs are held in memory
and if
> > > memory
> > > >>>> runs low, get swapped out to temp storage.
> > > >>>>
> > > >>>> > 2) If then I start feeding a 2nd queue, "Memory percent
used"
> > > >>>> continues
> > > >>>> > growing until it reaches 140%. So it looks like memoryUsage
does
> > not
> > > >>>> apply
> > > >>>> > globally, but on a per queue basis?
> > > >>>>
> > > >>>> What version of AMQ do you use? The sum of the memory usage
of all
> > > >>>> queues should not go any higher than the configured memoryUsage
> > > limit. If
> > > >>>> you're not on 5.5.1 or higher releases, then I suggest to
upgrade.
> > > >>>>
> > > >>>> > Using memoryLimit on the queue's policyEntry gives more
control
> > over
> > > >>>> this,
> > > >>>> > but it's just a variation, "Memory percent used" can
grow more
> > than
> > > >>>> 100%
> > > >>>> > anyway.
> > > >>>>
> > > >>>> With the default store cursor this should not be the case
from
> what
> > I
> > > >>>> know.
> > > >>>>
> > > >>>>
> > > >>>> >
> > > >>>> > 3) If #2 is true, then how would I prevent the broker
from
> running
> > > >>>> out of
> > > >>>> > memory in case queues would continue to be created?
> > > >>>>
> > > >>>> Just like above comment. I would expect the brokers
> > MemoryPercentUsage
> > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> remains
> > > fairly
> > > >>>> much at 70%.
> > > >>>> Not sure why you would see a different behavior? Using an
old
> > version
> > > >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> > > >>>> Could you perhaps also test with
> > > >>>>
> > > >>>> >
> > > >>>> >
> > > >>>> > Maybe I'm misunderstanding and some of these settings
make no
> > sense
> > > >>>> when
> > > >>>> > producerFlowControl is disabled?
> > > >>>> >
> > > >>>> > Thanks in advance.
> > > >>>> >
> > > >>>> > Juan
> > > >>>>
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Torsten Mielke
> > > >>>> torsten@fusesource.com
> > > >>>> tmielke.blogspot.com
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
> >
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message