activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Mielke <tors...@fusesource.com>
Subject Re: Understanding memoryUsage (once again!)
Date Mon, 19 Nov 2012 08:56:45 GMT
ActiveMQ 5.3 is fairly old now.
I would have though that 5.7 should honor the memoryUsage settings correctly. 
Unfortunately I don't have the time at the moment to test this myself. 
If you are certain that it also does not work in 5.7, perhaps raise a JIRA ticket on the ActiveMQ
project. Ideally, attach a little unit test that reproduces this behavior.

Regards,
Torsten Mielke
torsten@fusesource.com
tmielke.blogspot.com



On Nov 16, 2012, at 7:38 PM, Juan Nin wrote:

> nope, adding a 3rd queue the 3rd one also gets this same value, so even if
> it's the memory usage of the queue it's anyway going beyond..
> 
> 
> On Fri, Nov 16, 2012 at 4:32 PM, Juan Nin <juanin@gmail.com> wrote:
> 
>> Might it be just a bug on how the MemoryPercentUsage is calculated?
>> 
>> If I connect via JMX using console, I can see the MemoryPercentUsage as
>> 112 right now.
>> If I go to each of the 2 queues on them I see CursorMemoryUsage with value
>> 29360604, which would be 28mb each, summing a total of 56mb (just a bit
>> more than the specified memoryUsage of 50mb).
>> 
>> Not sure I'm interpreting these values correctly though, first time I
>> access it via jconsole...
>> 
>> 
>> On Fri, Nov 16, 2012 at 4:07 PM, Juan Nin <juanin@gmail.com> wrote:
>> 
>>> On that config there's a 40mb memoryLimit per queue, but also tested it
>>> without it with same results.
>>> 
>>> 
>>> On Fri, Nov 16, 2012 at 4:05 PM, Juan Nin <juanin@gmail.com> wrote:
>>> 
>>>> Hi Torsten!
>>>> 
>>>> I'm using ActiveMQ 5.3.2, but also tested it on 5.7.0 with the same
>>>> results...
>>>> This is my 5.3.2 config:
>>>> 
>>>> <beans
>>>>  xmlns="http://www.springframework.org/schema/beans"
>>>>  xmlns:amq="http://activemq.apache.org/schema/core"
>>>>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>  xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>  http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd">
>>>> 
>>>>    <bean
>>>> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>>        <property name="locations">
>>>> 
>>>> <value>file:${activemq.base}/conf/credentials.properties</value>
>>>>        </property>
>>>>    </bean>
>>>> 
>>>>    <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>>> destroyApplicationContextOnStop="true" advisorySupport="false">
>>>> 
>>>>        <destinationPolicy>
>>>>            <policyMap>
>>>>                <policyEntries>
>>>>                    <policyEntry topic=">" producerFlowControl="true"
>>>> memoryLimit="5mb">
>>>>                        <pendingSubscriberPolicy>
>>>>                            <vmCursor />
>>>>                        </pendingSubscriberPolicy>
>>>>                    </policyEntry>
>>>>                    <policyEntry queue=">" producerFlowControl="false"
>>>> optimizedDispatch="true" memoryLimit="40mb">
>>>>                <deadLetterStrategy>
>>>>                            <individualDeadLetterStrategy
>>>> queuePrefix="DLQ." useQueueForQueueMessages="true" />
>>>>                        </deadLetterStrategy>
>>>>                    </policyEntry>
>>>>                </policyEntries>
>>>>            </policyMap>
>>>>        </destinationPolicy>
>>>> 
>>>>        <managementContext>
>>>>            <managementContext connectorPort="2011"/>
>>>>        </managementContext>
>>>> 
>>>>        <persistenceAdapter>
>>>>            <kahaDB directory="${activemq.base}/data/kahadb"
>>>> enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
>>>> indexCacheSize="1000"/>
>>>>        </persistenceAdapter>
>>>> 
>>>>        <systemUsage>
>>>>            <systemUsage>
>>>>              <memoryUsage>
>>>>                    <memoryUsage limit="50 mb"/>
>>>>                 </memoryUsage>
>>>>                <storeUsage>
>>>>                    <storeUsage limit="1 gb" name="foo"/>
>>>>                </storeUsage>
>>>>                <tempUsage>
>>>>                    <tempUsage limit="3 gb"/>
>>>>                </tempUsage>
>>>>            </systemUsage>
>>>>        </systemUsage>
>>>> 
>>>>        <transportConnectors>
>>>>            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>>>> "/>
>>>>    <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
>>>>        </transportConnectors>
>>>> 
>>>>    </broker>
>>>> 
>>>>    <import resource="jetty.xml"/>
>>>> 
>>>> </beans>
>>>> 
>>>> 
>>>> Using just a simple PHP script with Stomp for feeding the queues
>>>> (running it twice with different queue name):
>>>> 
>>>> <?php
>>>> 
>>>> require_once("Stomp.php");
>>>> 
>>>> $amq = new Stomp("tcp://localhost:61613");
>>>> $amq->connect();
>>>> 
>>>> for($i=1; $i <= 100000; $i++)
>>>> {
>>>> if($i%1000 == 0)
>>>> {
>>>> echo "\nmsg #: $i";
>>>> }
>>>> $amq->send("/queue/test", "this is test message # $i"
>>>> ,array('persistent' => 'true'));
>>>> }
>>>> 
>>>> $amq->disconnect();
>>>> 
>>>> ?>
>>>> 
>>>> 
>>>> 
>>>> On Fri, Nov 16, 2012 at 3:47 PM, Torsten Mielke <torsten@fusesource.com>wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> See in-line response.
>>>>> 
>>>>> On Nov 16, 2012, at 6:29 PM, Juan Nin wrote:
>>>>> 
>>>>>> Hi!
>>>>>> 
>>>>>> After some heavy digging about Producer Flow control and the
>>>>> systemUsage
>>>>>> properties a couple of years ago, I thought I quite understood it.
>>>>>> But yesterday I found that one of my configs was not behaving exactly
>>>>> as I
>>>>>> expected, so started doing some tests, and I see certain behaviours
>>>>> which
>>>>>> don't seem to match what the docs and posts that I find on the list
or
>>>>>> other forums say.
>>>>>> 
>>>>>> "storeUsage" is perfectly clear, it's the max space that persistent
>>>>>> messages can use to be stored in disk.
>>>>>> "tempUsage"" applies to file cursors on non-persistent messages,
so
>>>>> as to
>>>>>> flush to disk if memory limits are reached (I don't care much about
>>>>> this
>>>>>> one anyway, I always use persistent messages).
>>>>> 
>>>>> Correct.
>>>>> 
>>>>>> 
>>>>>> Now, according to most posts, memoryUsage would be the maximum memory
>>>>> that
>>>>>> the broker would be available to use.
>>>>>> On this post:
>>>>>> 
>>>>> http://stackoverflow.com/questions/7646057/activemq-destinationpolicy-and-systemusage-configurationit
>>>>>> says that "memoryUsage corresponds to the amount of memory that's
>>>>>> assigned to the in-memory store".
>>>>> 
>>>>> Correct.
>>>>> 
>>>>>> 
>>>>>> For example, on my tests using the following config (only showing
>>>>> relevant
>>>>>> parts):
>>>>>> 
>>>>>> <policyEntry queue=">" producerFlowControl="false"
>>>>> optimizedDispatch="true">
>>>>>>   <deadLetterStrategy>
>>>>>>       <individualDeadLetterStrategy queuePrefix="DLQ."
>>>>>> useQueueForQueueMessages="true" />
>>>>>>   </deadLetterStrategy>
>>>>>> </policyEntry>
>>>>>> 
>>>>>> <systemUsage>
>>>>>>   <systemUsage>
>>>>>>       <memoryUsage>
>>>>>>           <memoryUsage limit="100 mb"/>
>>>>>>       </memoryUsage>
>>>>>>       <storeUsage>
>>>>>>           <storeUsage limit="1 gb" name="foo"/>
>>>>>>       </storeUsage>
>>>>>>       <tempUsage>
>>>>>>           <tempUsage limit="3 gb"/>
>>>>>>       </tempUsage>
>>>>>>   </systemUsage>
>>>>>> </systemUsage>
>>>>>> 
>>>>>> With that config I would expect the broker to use 100 mb of maximum
>>>>> memory
>>>>>> among all queues. So it could maybe use 30mb in one queue and 70mb
in
>>>>>> second queue.
>>>>>> 
>>>>>> 
>>>>>> 1) What I'm seeing is that if I start feeding a queue without
>>>>> consuming it,
>>>>>> the "Memory percent used" grows up to 70%, after that it doesn't
grow
>>>>>> anymore.
>>>>>> What is it doing exactly there? The first 70% is stored in memory
>>>>> (apart
>>>>>> from disk since it's persistent), and all the rest that continues
>>>>> being fed
>>>>>> goes just to disk?
>>>>> 
>>>>> This behavior is correct. For queues the default cursor is store
>>>>> cursor. It keeps any newly arrived msgs in memory as long as it does
not
>>>>> reach the configured memory limit (either configured on the queue per
>>>>> destination or globally in memoryUsage settings).
>>>>> Once the cursor reaches 70% of the configured limit (in your case of
>>>>> the memoryUsage limit since you don't specify a per-destination limit),
it
>>>>> will not keep any more messages in memory.
>>>>> Instead it will reload these messages from the store when its time to
>>>>> dispatch them. The broker anyway persists any msgs it receives before
>>>>> passing on to the cursor.
>>>>> This limit of 70% can be configured and raised to e..g 100%.
>>>>> This behavior is kind of an optimization. That way you run less often
>>>>> into producer-flow-control.
>>>>> As long as the persistence store is not running full, there is no need
>>>>> to block producers, since the cursor can also load the messages from
the
>>>>> store and does not necessarily have to keep them in memory.
>>>>> If you configure the vmQueueCursor, then the behavior is different.
>>>>> This cursor will not be able to load msgs to the store but needs to keep
>>>>> them all in memory. The vmQueueCursor used to be the default cursor in
>>>>> older version of AMQ.
>>>>> 
>>>>> Also note that topic msgs and non-persistent queue messages are not
>>>>> handled by the store cursor. These msgs are held in memory and if memory
>>>>> runs low, get swapped out to temp storage.
>>>>> 
>>>>>> 2) If then I start feeding a 2nd queue, "Memory percent used"
>>>>> continues
>>>>>> growing until it reaches 140%. So it looks like memoryUsage does
not
>>>>> apply
>>>>>> globally, but on a per queue basis?
>>>>> 
>>>>> What version of AMQ do you use? The sum of the memory usage of all
>>>>> queues should not go any higher than the configured memoryUsage limit.
If
>>>>> you're not on 5.5.1 or higher releases, then I suggest to upgrade.
>>>>> 
>>>>>> Using memoryLimit on the queue's policyEntry gives more control over
>>>>> this,
>>>>>> but it's just a variation, "Memory percent used" can grow more than
>>>>> 100%
>>>>>> anyway.
>>>>> 
>>>>> With the default store cursor this should not be the case from what I
>>>>> know.
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 3) If #2 is true, then how would I prevent the broker from running
>>>>> out of
>>>>>> memory in case queues would continue to be created?
>>>>> 
>>>>> Just like above comment. I would expect the brokers MemoryPercentUsage
>>>>> won't grow over 100% and the destinations MemoryPercentUsage remains
fairly
>>>>> much at 70%.
>>>>> Not sure why you would see a different behavior? Using an old version
>>>>> of AMQ perhaps? Or explicitly configuring for the vmQueueCursor?
>>>>> Could you perhaps also test with
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Maybe I'm misunderstanding and some of these settings make no sense
>>>>> when
>>>>>> producerFlowControl is disabled?
>>>>>> 
>>>>>> Thanks in advance.
>>>>>> 
>>>>>> Juan
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Torsten Mielke
>>>>> torsten@fusesource.com
>>>>> tmielke.blogspot.com
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 







Mime
View raw message