activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aliu <a...@endeca.com>
Subject Re: systemUsage configuration
Date Wed, 16 Jul 2008 16:01:45 GMT

Hi, just to clarify. In my last post, when I mentioned 'all the pending
messages are in memory', I actually mean uncommitted messages within a
transaction. Again, can someone confirm if this statement (all uncommitted
messages are kept in memory) is true?



aliu wrote:
> 
> The reason I've been trying to figure out these memory limit
> configurations is because we have the following use case:
> 
> I want to be able to send a large amount of data (>2g) to the queue
> without any consumer consuming it for some time, but it seems like active
> mq actually keeps all the pending messages in memory, I just want to
> confirm this observation since this would put a lot of constraint on our
> memory requirement.
> 
> I are using persistent messaging with transactions, with memoryUsage set
> to 1g, when sending over 50mb of data, I get an OutOfMemory error in the
> middle (around 40mb). But when I tried to do a commit with every 3mb of
> data, there was no errors. 
> 
> I just want to confirm whether active mq actually keeps all the pending
> messages in memory.
> 
> Thanks,
> Audrey
> 
> 
> 
> ammulder wrote:
>> 
>> Are you validating the XML just to make sure it's not a syntax error?
>> 
>> I get stuff onto disk with a configuration like this (and a producer
>> sending 1 MB messages):
>> 
>> <destinationPolicy>
>>     <policyMap>
>>         <policyEntries>
>>             <policyEntry queue=">" memoryLimit="50mb"/>
>>             <policyEntry topic=">" memoryLimit="50mb">
>> ...
>> <systemUsage>
>>     <systemUsage>
>>         <memoryUsage>
>>             <memoryUsage limit="20 mb"/>
>>         </memoryUsage>
>> ...
>> 
>> 
>> Thanks,
>>        Aaron
>> 
>> On Tue, Jul 15, 2008 at 10:50 AM, aliu <aliu@endeca.com> wrote:
>>>
>>> Hi,
>>> Thanks for the quick reply. If I don't want to keep everything in
>>> memory,
>>> and I want to be able to set how much should be stored in memory and if
>>> there are more stuff coming, it'd be spilled out to disk, what
>>> configuration
>>> settings can I use?
>>>
>>> I actually tried setting the destination limits high and the system
>>> limit
>>> low, with the following config:
>>> <policyEntry queue=">" memoryLimit="1 gb"/>
>>> ...
>>> <memoryUsage>
>>>    <memoryUsage limit="30 mb"/>
>>> </memoryUsage>
>>>
>>> When I'm trying to send 50mb worth of data, my producer blocks after
>>> sending
>>> about 30mb of data. And I also tried with destination limits low and
>>> system
>>> limit high, my producer also blocks after sending the lower limit of the
>>> two
>>> settings.
>>>
>>> Just wondering if anyone can give me some advice on what I can set on
>>> how
>>> much to be allowed in memory?
>>>
>>> Thanks in advance.
>>> Audrey
>>>
>>> ammulder wrote:
>>>>
>>>> I am by no means an authority on this, but I'm not sure the store
>>>> usage or temp usage settings are actually used for anything.
>>>>
>>>> I believe you want your memory usage setting to be high enough to hold
>>>> all the traffic you expect to be in memory at once.  I think this is
>>>> e.g. non-persistent messages that have not yet been delivered (regular
>>>> in-flight messages, queues that are backed up, topics with durable
>>>> subscribers who are away, etc.).  There are separate settings that can
>>>> be applied to the memory limit for each destination (or wildcard
>>>> groups of queues/topics, etc.).
>>>>
>>>> If you set the destination limits high and the system limit low, you
>>>> can cause ActiveMQ to start using disk space (as "swap" I guess), but
>>>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
>>>> bound by the store/temp usage settings.
>>>>
>>>> If you are up against memory limits (due to slow/disconnected
>>>> consumers or whatever), there are various additional considerations
>>>> and configuration options you may want to use.
>>>>
>>>> Thanks,
>>>>       Aaron
>>>>
>>>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <aliu@endeca.com> wrote:
>>>>>
>>>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how
>>>>> to
>>>>> tweak different memory settings.
>>>>> I read on the forum that storeUsage controls the maximum size of the
>>>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>>>>> broker
>>>>> will use.
>>>>>
>>>>> However, when I ran a test of sending 50,000 messages with 1k for each
>>>>> message, and I've used the following config:
>>>>>         <systemUsage>
>>>>>            <systemUsage>
>>>>>                <memoryUsage>
>>>>>                    <memoryUsage limit="30 mb"/>
>>>>>                </memoryUsage>
>>>>>                <storeUsage>
>>>>>                    <storeUsage limit="1 gb" name="foo"/>
>>>>>                </storeUsage>
>>>>>                <tempUsage>
>>>>>                    <tempUsage limit="1 mb"/>
>>>>>                </tempUsage>
>>>>>            </systemUsage>
>>>>>        </systemUsage>
>>>>>
>>>>> My producer would block after sending about 30mb worth of data, even
>>>>> though
>>>>> I have specified the storeUsage to be big enough to contain all my
>>>>> messages.
>>>>> I thought what it would be is to have 30mb worth of data in memory,
>>>>> then
>>>>> it'll spill over once it exceeds the limit.
>>>>>
>>>>> Can someone please tell me in detail how these settings work together?
>>>>> I
>>>>> am
>>>>> at a loss here.
>>>>>
>>>>> I am using 5.1, persistent messaging with transactions.
>>>>>
>>>>> Thanks in advance.
>>>>> Audrey
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18490842.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message