activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aliu <a...@endeca.com>
Subject Re: systemUsage configuration
Date Tue, 15 Jul 2008 19:31:44 GMT

The reason I've been trying to figure out these memory limit configurations
is because we have the following use case:

I want to be able to send a large amount of data (>2g) to the queue without
any consumer consuming it for some time, but it seems like active mq
actually keeps all the pending messages in memory, I just want to confirm
this observation since this would put a lot of constraint on our memory
requirement.

I are using persistent messaging with transactions, with memoryUsage set to
1g, when sending over 50mb of data, I get an OutOfMemory error in the middle
(around 40mb). But when I tried to do a commit with every 3mb of data, there
was no errors. 

I just want to confirm whether active mq actually keeps all the pending
messages in memory.

Thanks,
Audrey



ammulder wrote:
> 
> Are you validating the XML just to make sure it's not a syntax error?
> 
> I get stuff onto disk with a configuration like this (and a producer
> sending 1 MB messages):
> 
> <destinationPolicy>
>     <policyMap>
>         <policyEntries>
>             <policyEntry queue=">" memoryLimit="50mb"/>
>             <policyEntry topic=">" memoryLimit="50mb">
> ...
> <systemUsage>
>     <systemUsage>
>         <memoryUsage>
>             <memoryUsage limit="20 mb"/>
>         </memoryUsage>
> ...
> 
> 
> Thanks,
>        Aaron
> 
> On Tue, Jul 15, 2008 at 10:50 AM, aliu <aliu@endeca.com> wrote:
>>
>> Hi,
>> Thanks for the quick reply. If I don't want to keep everything in memory,
>> and I want to be able to set how much should be stored in memory and if
>> there are more stuff coming, it'd be spilled out to disk, what
>> configuration
>> settings can I use?
>>
>> I actually tried setting the destination limits high and the system limit
>> low, with the following config:
>> <policyEntry queue=">" memoryLimit="1 gb"/>
>> ...
>> <memoryUsage>
>>    <memoryUsage limit="30 mb"/>
>> </memoryUsage>
>>
>> When I'm trying to send 50mb worth of data, my producer blocks after
>> sending
>> about 30mb of data. And I also tried with destination limits low and
>> system
>> limit high, my producer also blocks after sending the lower limit of the
>> two
>> settings.
>>
>> Just wondering if anyone can give me some advice on what I can set on how
>> much to be allowed in memory?
>>
>> Thanks in advance.
>> Audrey
>>
>> ammulder wrote:
>>>
>>> I am by no means an authority on this, but I'm not sure the store
>>> usage or temp usage settings are actually used for anything.
>>>
>>> I believe you want your memory usage setting to be high enough to hold
>>> all the traffic you expect to be in memory at once.  I think this is
>>> e.g. non-persistent messages that have not yet been delivered (regular
>>> in-flight messages, queues that are backed up, topics with durable
>>> subscribers who are away, etc.).  There are separate settings that can
>>> be applied to the memory limit for each destination (or wildcard
>>> groups of queues/topics, etc.).
>>>
>>> If you set the destination limits high and the system limit low, you
>>> can cause ActiveMQ to start using disk space (as "swap" I guess), but
>>> I hear that's prone to deadlocks and etc. -- plus it doesn't seem
>>> bound by the store/temp usage settings.
>>>
>>> If you are up against memory limits (due to slow/disconnected
>>> consumers or whatever), there are various additional considerations
>>> and configuration options you may want to use.
>>>
>>> Thanks,
>>>       Aaron
>>>
>>> On Mon, Jul 14, 2008 at 6:47 PM, aliu <aliu@endeca.com> wrote:
>>>>
>>>> Hi, I am a new user of the ActiveMQ and I was trying to figure out how
>>>> to
>>>> tweak different memory settings.
>>>> I read on the forum that storeUsage controls the maximum size of the
>>>> AMQMessageStore, and memoryUsage is the maximum amount of memory the
>>>> broker
>>>> will use.
>>>>
>>>> However, when I ran a test of sending 50,000 messages with 1k for each
>>>> message, and I've used the following config:
>>>>         <systemUsage>
>>>>            <systemUsage>
>>>>                <memoryUsage>
>>>>                    <memoryUsage limit="30 mb"/>
>>>>                </memoryUsage>
>>>>                <storeUsage>
>>>>                    <storeUsage limit="1 gb" name="foo"/>
>>>>                </storeUsage>
>>>>                <tempUsage>
>>>>                    <tempUsage limit="1 mb"/>
>>>>                </tempUsage>
>>>>            </systemUsage>
>>>>        </systemUsage>
>>>>
>>>> My producer would block after sending about 30mb worth of data, even
>>>> though
>>>> I have specified the storeUsage to be big enough to contain all my
>>>> messages.
>>>> I thought what it would be is to have 30mb worth of data in memory,
>>>> then
>>>> it'll spill over once it exceeds the limit.
>>>>
>>>> Can someone please tell me in detail how these settings work together?
>>>> I
>>>> am
>>>> at a loss here.
>>>>
>>>> I am using 5.1, persistent messaging with transactions.
>>>>
>>>> Thanks in advance.
>>>> Audrey
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/systemUsage-configuration-tp18454843p18454843.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/systemUsage-configuration-tp18454843p18467144.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/systemUsage-configuration-tp18454843p18473128.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message