activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cybexion <cybex...@email.de>
Subject Re: Performance in relation to queue size
Date Fri, 30 Jan 2009 09:40:51 GMT

I'm not sure, if this is all related to the Memoryleak I found in AMQ 5.2
see
http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-to21722796.html

I just made a new test run last night and my PS Old Gen is about to explode.
GC has no effect any more.





Cybexion wrote:
> 
> Hi,
> 
> I tried all this without success.
> The more messages are in the queue the slower it gets. My system was
> running during the whole weekend now. The first hour I had about 900
> messages per second (JMS sending took about 0ms). Then after about 1-2
> hours JMS sending suddently takes 1 ore more seconds.
> 
> My config looks like this (just the interesting stuff):
>         <bean id="store"
> class="org.apache.activemq.store.amq.AMQPersistenceAdapter">
>                 <property name="directory" value="${activemq.base}/data"/>
>                 <!-- 1GB = 1073741824 -->
>                 <property name="maxFileLength"         
> value="1073741824"/>
>                 <property name="maxReferenceFileLength"
> value="104857600"/>
>                 <property name="checkpointInterval"     value="60000"/>
>                 <property name="cleanupInterval"        value="30000"/>
>                 <property name="persistentIndex"        value="true"/>
>                 <property name="indexMaxBinSize"        value="10485760"/>
>                 <property name="indexBinSize"           value="1048576"/>
>                 <property name="indexPageSize"          value="65536"/>
>         </bean>
> 
>   <broker xmlns="http://activemq.apache.org/schema/core"
>         persistent="true"
>         advisorySupport="false"
>         deleteAllMessagesOnStartup="false"
>         useJmx="true"
>         monitorConnectionSplits="false"
>         splitSystemUsageForProducersConsumers="false"
>         brokerName="localhost"
>         dataDirectory="${activemq.base}/data"
>         persistenceAdapter="#store"
>         >
> 
> 
>        <!--  The maximum about of space the broker will use before slowing
> down producers -->
>         <systemUsage>
>             <systemUsage sendFailIfNoSpace="true">
>                 <memoryUsage>
>                     <memoryUsage limit="1gb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="30gb" store="#store"/>
>                 </storeUsage>
> 
>                 <!-- temp usage is used with non persistent messages only
> -->
>                 <tempUsage>
>                     <tempUsage limit="64mb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
> 
> Anyone else having an idea?
> Next thing I try is to limit the queue size to lets say on GB and see if
> performance is fine if the size is limited.
> 
> 
> 
> 
> 
> 
> Gary Tully wrote:
>> 
>>> Well, the main question is now:
>>> 1. Why is the queue getting slower when there are a lot of messages
>>> inside?
>> So with X million messages, enqueue times are 4-7 seconds?
>> This is probabally because the hashIndex is limited, I think you need
>> to make the indexMaxBinSize quite large and use the same value for
>> indexBinSize so that no resizing takes place at runtime.  This should
>> speed up index insertion.
>> Also increasing the indexPageSize may help by increasing the quantity
>> of data persisted in one write.
>> 
>> After that, this may be indicative of a problem with the single large
>> hash index file.
>> 
>> 
>> 
>>> 2. How can I change the size of the data-queue-data-959 files?
>> The size of the files in the reference store are controlled by the
>> maxReferenceFileLength attribute.
>> 
>>> Would it make sense to have a few big of those files instead of a lot of
>>> small one?
>> Not sure, I think this is only relevant to cleanup, a large file with
>> lots of references may take a long time to be unferenced and ready for
>> cleanup.
>> 
>>> 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking
>> This is the persistent part of the persistent hashIndex
>> 
>> The following link gives a overview of the amq persistence adapter
>> explaining the layout etc:
>> http://activemq.apache.org/amq-message-store.html
>> 
>> 2009/1/23 Cybexion <cybexion@email.de>:
>>>
>>> Hello,
>>>
>>> I changed my AMQPersistenceAdapter to write 1 GB files (total store size
>>> is
>>> 30GB)
>>>        <bean id="store"
>>> class="org.apache.activemq.store.amq.AMQPersistenceAdapter">
>>>                <property name="directory"
>>> value="${activemq.base}/data"/>
>>>                <!-- 1GB = 1073741824 -->
>>>                <property name="maxFileLength" value="1073741824"/>
>>>                <property name="checkpointInterval" value="60000"/>
>>>                <property name="cleanupInterval" value="30000"/>
>>>                <property name="persistentIndex" value="true"/>
>>>        </bean>
>>>
>>> After that I created messages during the whole night into the queue.
>>> The result looks like this:
>>>
>>> ls -l data/journal
>>> -rw-r--r-- 1 activemq java 1073741777 2009-01-22 21:06 data-2
>>> -rw-r--r-- 1 activemq java 1073741637 2009-01-22 23:17 data-3
>>> -rw-r--r-- 1 activemq java 1073741573 2009-01-23 01:17 data-4
>>> -rw-r--r-- 1 activemq java 1073741815 2009-01-23 03:40 data-5
>>> -rw-r--r-- 1 activemq java 1073741674 2009-01-23 06:48 data-6
>>> -rw-r--r-- 1 activemq java 1073217536 2009-01-23 11:22 data-7
>>> -rw-r--r-- 1 activemq java       2088 2009-01-23 11:22 data-control
>>>
>>> About 5.2 GB of data has been created.
>>>
>>> These are the hundreds of files containing the keys to the messages.
>>> ls -l data/kr-store/data
>>> -rw-r--r-- 1 activemq java    2097142 2009-01-23 11:04
>>> data-queue-data-957
>>> -rw-r--r-- 1 activemq java    2097117 2009-01-23 11:11
>>> data-queue-data-958
>>> -rw-r--r-- 1 activemq java    2068877 2009-01-23 11:21
>>> data-queue-data-959
>>> -rw-r--r-- 1 activemq java 1462429697 2009-01-23 11:24
>>> hash-index-queue-data_queue#3a#2f#2ftracking
>>> -rw-r--r-- 1 activemq java        153 2009-01-22 17:10
>>> index-container-roots
>>> -rw-r--r-- 1 activemq java  705609480 2009-01-23 11:24 index-queue-data
>>> -rw-r--r-- 1 activemq java          0 2009-01-22 17:09 lock
>>>
>>> ls -l | wc
>>>    721    5762   54026
>>>
>>> Here are some additional JMX infos about the broker
>>>
>>> ActiveMQ Broker Memory percent Usage    :       0,00
>>> ActiveMQ Broker Store percent Usage     :       26,00
>>> ActiveMQ Broker Temp percent Usage      :       0,00
>>> ActiveMQ Broker Consumer Count  :       1,00
>>> ActiveMQ Broker Total Dequeue Count     :       3.629.262,00
>>> ActiveMQ Broker Total Enqueue Count     :       16.132.693,00
>>> ActiveMQ Broker Total Message Count     :       12.503.432,00
>>>
>>> Now to the bad news:
>>> It looks like the more messages are in the queue, the slower it gets to
>>> send
>>> new messages into it (using JmsTemplate.send() method).
>>> When I currently run I java programm that inserts messages into the
>>> Queue I
>>> have response times of 4-7 seconds !!!!
>>> This is not good.
>>> Here is an extract of the AMQ log file:
>>> 2009-01-22 18:08:11,229 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 1024 to 2048
>>> 2009-01-22 18:10:24,642 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 2048 to 4096
>>> 2009-01-22 18:13:35,224 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 4096 to 8192
>>> 2009-01-22 18:18:12,564 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 8192 to 16384
>>>
>>> Well, the main question is now:
>>> 1. Why is the queue getting slower when there are a lot of messages
>>> inside?
>>> 2. How can I change the size of the data-queue-data-959 files?
>>> Would it make sense to have a few big of those files instead of a lot of
>>> small one?
>>> 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking
>>>
>>> Thanks and regards
>>>
>>> Jörg
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21622121.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> 
>> -- 
>> http://blog.garytully.com
>> 
>> Open Source SOA
>> http://FUSESource.com
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21743840.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message