activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Nin <jua...@gmail.com>
Subject Re: Understanding memoryUsage (once again!)
Date Thu, 29 Nov 2012 19:45:43 GMT
Hi Christian!

Yes, actually that's what I'm doing, just setting per destination policies
which work for me.
I anyway needed them because I'm creating queues with lots of messages,
which won't be immediately consumed, so having them store a lot into memory
ended up slowing things up.

So I just assigned enough memory to the broker so as not run into issues.

Thanks again.


On Tue, Nov 27, 2012 at 9:40 PM, Christian Posta
<christian.posta@gmail.com>wrote:

> See inline...
>
>
> On Wed, Nov 21, 2012 at 12:04 PM, Juan Nin <juanin@gmail.com> wrote:
>
> > Hi!
> >
> > Sorry for the delay in replying, buried on a project.
> >
> > As I mentioned before, I had tested this with 5.7.0 with the same
> > behaviour.
> > I just tested it again (both with 5.3.2 and 5.7.0) and same thing, and on
> > my case it doesn't matter if there are consumers or not, it always seems
> to
> > make usage of the memory.
> >
> > Although I guess in theory that should not affect, did you use Stomp for
> > your testing, or maybe you used Openwire?
> > I'm using Stomp for my testing.
> >
> > Might be though that the broker's memory itself is not going beyond 70%
> of
> > memoryUsage, but this is just per destination counters as you mentioned.
> > In which case I guess the value shown as "Memory percent used" is a bit
> > confusing... But haven't had much time to really test the possibility of
> > exhausting the broker's memory.
> >
> No, i believe what you're seeing is correct. The broker's memory limit is
> going beyond memoryUsage (way beyond). When a queue checks whether memory
> is full, it will only do something interesting if producer flow control is
> enabled. Otherwise, it will continue on. You are seeing that it will
> continue to add messages until the Queue's memory limit (40MB) reaches the
> 70% mark. Since MemoryUsages are hierarchical, this means it will also
> account for messages in the overall broker memory as well. For each queue,
> you'll see that it will continue to hold 70% of 40MB of memory. What you
> want in this case (if there are no consumers, or slow consumers) is to
> raise your system usage memory limit OR lower your per-destination limits
> OR lower your cursor highwatermark or a combination of all three.
>
> http://activemq.apache.org/per-destination-policies.html
>
> With PFC turned off, you're essentially telling the broker to take the
> message no matter what. There is a point at which you will run out of
> resources (memory, disk, etc). The trick is to find your use case and tune
> for that.
>
>
> >
> > Will try to do some more testing soon...
> >
> > Thanks
> >
> >
> > On Wed, Nov 21, 2012 at 2:28 PM, Christian Posta
> > <christian.posta@gmail.com>wrote:
> >
> > > Can you please try on 5.7?
> > > I just tried a test, and if there are no consumers to the queue then
> the
> > > memory usage will stay at 0%. The message will not be retained, ie, it
> > will
> > > be put into the store and kept there. If I add a consumer, and not try
> to
> > > consume, the message will be kept around in memory up to the cursor
> high
> > > watermark (70 by default).
> > >
> > > As I add more queues the same behavior as described above will happen.
> > If I
> > > attach consumers to the queues without consuming them (so no messages
> are
> > > consumed), then messages are kept in the cursor up to the high-water
> > > mark... note.. the high-water mark is relative to the
> > Destination/Cursor's
> > > MemoryUsage, not the global memory usage.
> > >
> > > If I continue adding queues, and with producer flow control set to
> > false, I
> > > too will see the *Global* memory usage go much higher than 100%. This
> is
> > > not surprising though, because as I understand, these usage memory
> > objects
> > > are really just counters. They don't enforce anything. When coupled
> with
> > > producer flow control, they can be used to determine when to enable
> PFC.
> > If
> > > PFC is false, it's up to the cursor to determine when to flush out to
> > disk.
> > > But each destination/cursor will have it's own system usage (with the
> > > global as the parent).
> > >
> > > Hope this helps. Can you please try with 5.7 and give us a report back?
> > > Thanks,
> > > Christian
> > >
> > >
> > >
> > > On Fri, Nov 16, 2012 at 11:38 AM, Juan Nin <juanin@gmail.com> wrote:
> > >
> > > > nope, adding a 3rd queue the 3rd one also gets this same value, so
> even
> > > if
> > > > it's the memory usage of the queue it's anyway going beyond..
> > > >
> > > >
> > > > On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <juanin@gmail.com> wrote:
> > > >
> > > > > Might it be just a bug on how the MemoryPercentUsage is calculated?
> > > > >
> > > > > If I connect via JMX using console, I can see the
> MemoryPercentUsage
> > as
> > > > > 112 right now.
> > > > > If I go to each of the 2 queues on them I see CursorMemoryUsage
> with
> > > > value
> > > > > 29360604, which would be 28mb each, summing a total of 56mb (just
a
> > bit
> > > > > more than the specified memoryUsage of 50mb).
> > > > >
> > > > > Not sure I'm interpreting these values correctly though, first
> time I
> > > > > access it via jconsole...
> > > > >
> > > > >
> > > > > On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <juanin@gmail.com>
> wrote:
> > > > >
> > > > >> On that config there's a 40mb memoryLimit per queue, but also
> tested
> > > it
> > > > >> without it with same results.
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <juanin@gmail.com>
> wrote:
> > > > >>
> > > > >>> Hi Torsten!
> > > > >>>
> > > > >>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with
the
> same
> > > > >>> results...
> > > > >>> This is my 5.3.2 config:
> > > > >>>
> > > > >>> <beans
> > > > >>>   xmlns="http://www.springframework.org/schema/beans"
> > > > >>>   xmlns:amq="http://activemq.apache.org/schema/core"
> > > > >>>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> > > > >>>   xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> > > > >>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> > > > >>>   http://activemq.apache.org/schema/core
> > > > >>> http://activemq.apache.org/schema/core/activemq-core.xsd">
> > > > >>>
> > > > >>>     <bean
> > > > >>>
> > > >
> > >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> > > > >>>         <property name="locations">
> > > > >>>
> > > > >>> <value>file:${activemq.base}/conf/credentials.properties</value>
> > > > >>>         </property>
> > > > >>>     </bean>
> > > > >>>
> > > > >>>     <broker xmlns="http://activemq.apache.org/schema/core"
> > > > >>> brokerName="localhost" dataDirectory="${activemq.base}/data"
> > > > >>> destroyApplicationContextOnStop="true" advisorySupport="false">
> > > > >>>
> > > > >>>         <destinationPolicy>
> > > > >>>             <policyMap>
> > > > >>>                 <policyEntries>
> > > > >>>                     <policyEntry topic=">"
> > producerFlowControl="true"
> > > > >>> memoryLimit="5mb">
> > > > >>>                         <pendingSubscriberPolicy>
> > > > >>>                             <vmCursor />
> > > > >>>                         </pendingSubscriberPolicy>
> > > > >>>                     </policyEntry>
> > > > >>>                     <policyEntry queue=">"
> > > producerFlowControl="false"
> > > > >>> optimizedDispatch="true" memoryLimit="40mb">
> > > > >>>                 <deadLetterStrategy>
> > > > >>>                             <individualDeadLetterStrategy
> > > > >>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
> > > > >>>                         </deadLetterStrategy>
> > > > >>>                     </policyEntry>
> > > > >>>                 </policyEntries>
> > > > >>>             </policyMap>
> > > > >>>         </destinationPolicy>
> > > > >>>
> > > > >>>         <managementContext>
> > > > >>>             <managementContext connectorPort="2011"/>
> > > > >>>         </managementContext>
> > > > >>>
> > > > >>>         <persistenceAdapter>
> > > > >>>             <kahaDB directory="${activemq.base}/data/kahadb"
> > > > >>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
> > > > >>> indexCacheSize="1000"/>
> > > > >>>         </persistenceAdapter>
> > > > >>>
> > > > >>>         <systemUsage>
> > > > >>>             <systemUsage>
> > > > >>>               <memoryUsage>
> > > > >>>                     <memoryUsage limit="50 mb"/>
> > > > >>>                  </memoryUsage>
> > > > >>>                 <storeUsage>
> > > > >>>                     <storeUsage limit="1 gb" name="foo"/>
> > > > >>>                 </storeUsage>
> > > > >>>                 <tempUsage>
> > > > >>>                     <tempUsage limit="3 gb"/>
> > > > >>>                 </tempUsage>
> > > > >>>             </systemUsage>
> > > > >>>         </systemUsage>
> > > > >>>
> > > > >>>         <transportConnectors>
> > > > >>>             <transportConnector name="openwire" uri="tcp://
> > > > 0.0.0.0:61616
> > > > >>> "/>
> > > > >>>     <transportConnector name="stomp" uri="stomp://0.0.0.0:61613
> "/>
> > > > >>>         </transportConnectors>
> > > > >>>
> > > > >>>     </broker>
> > > > >>>
> > > > >>>     <import resource="jetty.xml"/>
> > > > >>>
> > > > >>> </beans>
> > > > >>>
> > > > >>>
> > > > >>> Using just a simple PHP script with Stomp for feeding the
queues
> > > > >>> (running it twice with different queue name):
> > > > >>>
> > > > >>> <?php
> > > > >>>
> > > > >>> require_once("Stomp.php");
> > > > >>>
> > > > >>> $amq = new Stomp("tcp://localhost:61613");
> > > > >>> $amq->connect();
> > > > >>>
> > > > >>> for($i=1; $i <= 100000; $i++)
> > > > >>> {
> > > > >>> if($i%1000 == 0)
> > > > >>> {
> > > > >>>  echo "\nmsg #: $i";
> > > > >>> }
> > > > >>> $amq->send("/queue/test", "this is test message # $i"
> > > > >>> ,array('persistent' => 'true'));
> > > > >>> }
> > > > >>>
> > > > >>> $amq->disconnect();
> > > > >>>
> > > > >>> ?>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <
> > > > torsten@fusesource.com>wrote:
> > > > >>>
> > > > >>>> Hello,
> > > > >>>>
> > > > >>>> See in-line response.
> > > > >>>>
> > > > >>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
> > > > >>>>
> > > > >>>> > Hi!
> > > > >>>> >
> > > > >>>> > After some heavy digging about Producer Flow control
and the
> > > > >>>> systemUsage
> > > > >>>> > properties a couple of years ago, I thought I quite
understood
> > it.
> > > > >>>> > But yesterday I found that one of my configs was
not behaving
> > > > exactly
> > > > >>>> as I
> > > > >>>> > expected, so started doing some tests, and I see
certain
> > > behaviours
> > > > >>>> which
> > > > >>>> > don't seem to match what the docs and posts that
I find on the
> > > list
> > > > or
> > > > >>>> > other forums say.
> > > > >>>> >
> > > > >>>> > "storeUsage" is perfectly clear, it's the max space
that
> > > persistent
> > > > >>>> > messages can use to be stored in disk.
> > > > >>>> > "tempUsage"" applies to file cursors on non-persistent
> messages,
> > > so
> > > > >>>> as to
> > > > >>>> > flush to disk if memory limits are reached (I don't
care much
> > > about
> > > > >>>> this
> > > > >>>> > one anyway, I always use persistent messages).
> > > > >>>>
> > > > >>>> Correct.
> > > > >>>>
> > > > >>>> >
> > > > >>>> > Now, according to most posts, memoryUsage would
be the maximum
> > > > memory
> > > > >>>> that
> > > > >>>> > the broker would be available to use.
> > > > >>>> > On this post:
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
> > > > >>>> > says that "memoryUsage corresponds to the amount
of memory
> > that's
> > > > >>>> > assigned to the in-memory store".
> > > > >>>>
> > > > >>>> Correct.
> > > > >>>>
> > > > >>>> >
> > > > >>>> > For example, on my tests using the following config
(only
> > showing
> > > > >>>> relevant
> > > > >>>> > parts):
> > > > >>>> >
> > > > >>>> > <policyEntry queue=">" producerFlowControl="false"
> > > > >>>> optimizedDispatch="true">
> > > > >>>> >    <deadLetterStrategy>
> > > > >>>> >        <individualDeadLetterStrategy queuePrefix="DLQ."
> > > > >>>> > useQueueForQueueMessages="true" />
> > > > >>>> >    </deadLetterStrategy>
> > > > >>>> > </policyEntry>
> > > > >>>> >
> > > > >>>> > <systemUsage>
> > > > >>>> >    <systemUsage>
> > > > >>>> >        <memoryUsage>
> > > > >>>> >            <memoryUsage limit="100 mb"/>
> > > > >>>> >        </memoryUsage>
> > > > >>>> >        <storeUsage>
> > > > >>>> >            <storeUsage limit="1 gb" name="foo"/>
> > > > >>>> >        </storeUsage>
> > > > >>>> >        <tempUsage>
> > > > >>>> >            <tempUsage limit="3 gb"/>
> > > > >>>> >        </tempUsage>
> > > > >>>> >    </systemUsage>
> > > > >>>> > </systemUsage>
> > > > >>>> >
> > > > >>>> > With that config I would expect the broker to use
100 mb of
> > > maximum
> > > > >>>> memory
> > > > >>>> > among all queues. So it could maybe use 30mb in
one queue and
> > 70mb
> > > > in
> > > > >>>> > second queue.
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > 1) What I'm seeing is that if I start feeding a
queue without
> > > > >>>> consuming it,
> > > > >>>> > the "Memory percent used" grows up to 70%, after
that it
> doesn't
> > > > grow
> > > > >>>> > anymore.
> > > > >>>> > What is it doing exactly there? The first 70% is
stored in
> > memory
> > > > >>>> (apart
> > > > >>>> > from disk since it's persistent), and all the rest
that
> > continues
> > > > >>>> being fed
> > > > >>>> > goes just to disk?
> > > > >>>>
> > > > >>>> This behavior is correct. For queues the default cursor
is store
> > > > >>>> cursor. It keeps any newly arrived msgs in memory as
long as it
> > does
> > > > not
> > > > >>>> reach the configured memory limit (either configured
on the
> queue
> > > per
> > > > >>>> destination or globally in memoryUsage settings).
> > > > >>>> Once the cursor reaches 70% of the configured limit (in
your
> case
> > of
> > > > >>>> the memoryUsage limit since you don't specify a per-destination
> > > > limit), it
> > > > >>>> will not keep any more messages in memory.
> > > > >>>> Instead it will reload these messages from the store
when its
> time
> > > to
> > > > >>>> dispatch them. The broker anyway persists any msgs it
receives
> > > before
> > > > >>>> passing on to the cursor.
> > > > >>>> This limit of 70% can be configured and raised to e..g
100%.
> > > > >>>> This behavior is kind of an optimization. That way you
run less
> > > often
> > > > >>>> into producer-flow-control.
> > > > >>>> As long as the persistence store is not running full,
there is
> no
> > > need
> > > > >>>> to block producers, since the cursor can also load the
messages
> > from
> > > > the
> > > > >>>> store and does not necessarily have to keep them in memory.
> > > > >>>> If you configure the vmQueueCursor, then the behavior
is
> > different.
> > > > >>>> This cursor will not be able to load msgs to the store
but needs
> > to
> > > > keep
> > > > >>>> them all in memory. The vmQueueCursor used to be the
default
> > cursor
> > > in
> > > > >>>> older version of AMQ.
> > > > >>>>
> > > > >>>> Also note that topic msgs and non-persistent queue messages
are
> > not
> > > > >>>> handled by the store cursor. These msgs are held in memory
and
> if
> > > > memory
> > > > >>>> runs low, get swapped out to temp storage.
> > > > >>>>
> > > > >>>> > 2) If then I start feeding a 2nd queue, "Memory
percent used"
> > > > >>>> continues
> > > > >>>> > growing until it reaches 140%. So it looks like
memoryUsage
> does
> > > not
> > > > >>>> apply
> > > > >>>> > globally, but on a per queue basis?
> > > > >>>>
> > > > >>>> What version of AMQ do you use? The sum of the memory
usage of
> all
> > > > >>>> queues should not go any higher than the configured memoryUsage
> > > > limit. If
> > > > >>>> you're not on 5.5.1 or higher releases, then I suggest
to
> upgrade.
> > > > >>>>
> > > > >>>> > Using memoryLimit on the queue's policyEntry gives
more
> control
> > > over
> > > > >>>> this,
> > > > >>>> > but it's just a variation, "Memory percent used"
can grow more
> > > than
> > > > >>>> 100%
> > > > >>>> > anyway.
> > > > >>>>
> > > > >>>> With the default store cursor this should not be the
case from
> > what
> > > I
> > > > >>>> know.
> > > > >>>>
> > > > >>>>
> > > > >>>> >
> > > > >>>> > 3) If #2 is true, then how would I prevent the broker
from
> > running
> > > > >>>> out of
> > > > >>>> > memory in case queues would continue to be created?
> > > > >>>>
> > > > >>>> Just like above comment. I would expect the brokers
> > > MemoryPercentUsage
> > > > >>>> won't grow over 100% and the destinations MemoryPercentUsage
> > remains
> > > > fairly
> > > > >>>> much at 70%.
> > > > >>>> Not sure why you would see a different behavior? Using
an old
> > > version
> > > > >>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
> > > > >>>> Could you perhaps also test with
> > > > >>>>
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > Maybe I'm misunderstanding and some of these settings
make no
> > > sense
> > > > >>>> when
> > > > >>>> > producerFlowControl is disabled?
> > > > >>>> >
> > > > >>>> > Thanks in advance.
> > > > >>>> >
> > > > >>>> > Juan
> > > > >>>>
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>>
> > > > >>>> Torsten Mielke
> > > > >>>> torsten@fusesource.com
> > > > >>>> tmielke.blogspot.com
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Christian Posta*
> > > http://www.christianposta.com/blog
> > > twitter: @christianposta
> > >
> >
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message