Return-Path: 
 <users-return-17709-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org>
Delivered-To: apmail-activemq-users-archive@www.apache.org
Received: (qmail 4994 invoked from network); 26 Jan 2009 21:25:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 26 Jan 2009 21:25:03 -0000
Received: (qmail 56281 invoked by uid 500); 26 Jan 2009 21:25:02 -0000
Delivered-To: apmail-activemq-users-archive@activemq.apache.org
Received: (qmail 56040 invoked by uid 500); 26 Jan 2009 21:25:01 -0000
Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@activemq.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@activemq.apache.org>
List-Post: <mailto:users@activemq.apache.org>
List-Id: <users.activemq.apache.org>
Reply-To: users@activemq.apache.org
Delivered-To: mailing list users@activemq.apache.org
Received: (qmail 56029 invoked by uid 99); 26 Jan 2009 21:25:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jan 2009 13:25:01 -0800
X-ASF-Spam-Status: No, hits=2.6 required=10.0
	tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jan 2009 21:24:53 +0000
Received: from isper.nabble.com ([192.168.236.156])
	by kuber.nabble.com with esmtp (Exim 4.63)
	(envelope-from <lists@nabble.com>)
	id 1LRYwb-0000yy-9S
	for users@activemq.apache.org; Mon, 26 Jan 2009 13:24:33 -0800
Message-ID: <21673976.post@talk.nabble.com>
Date: Mon, 26 Jan 2009 13:24:33 -0800 (PST)
From: Cybexion <cybexion@email.de>
To: users@activemq.apache.org
Subject: Re: Performance in relation to queue size
In-Reply-To: <21601585.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Nabble-From: cybexion@email.de
References: <21601585.post@talk.nabble.com>
X-Virus-Checked: Checked by ClamAV on apache.org


Hello,

I just made another test:
What happens, if the Queue is full. Will the performance grow again?
I Changed my Queue size to GB (<storeUsage limit=3D"2gb" store=3D"#store"/>=
 ).
Now I restarted my test, having 900 JMS Producers sending messages into the
Queue.
The first few minutes it is very fast. After a while JMS sendings goes
slower and slower.
Whenn the Queue is about to reach her maximum size I can only send about 10=
0
JMS Requests per second (if at all).
But when the Queue has reached the maximum size, and
ResourceAllocationException are thrown on the server side like this:

2009-01-26 22:09:40,797 [127.0.0.1:48276] ERROR
emq.broker.TransportConnection - Async error occurred:
javax.jms.ResourceAllocationE
xception: Usage Manager Store is Full
javax.jms.ResourceAllocationException: Usage Manager Store is Full

Then the performance goes up again.
Currently I can see again about 900 Messages per second.
An interesting point here is, that the Queue size is shrinking but the
Broker Store percent usage is increating. I'm now at about 104% store usage=
.
Btw. I would be nice if one could change the exception above from ERROR to
warning. In my case this is not an ERROR and I would like to not see it in
my logs.

Concerning the performance I have the following theory:
While the Queue is growing there is a file that is constantly resizing (or
copied?) This causes more and more delay th bigger the queue gets.
I don't think it is one of the data file from the journal directory.
In my case they look like this:

ls -l data/journal
-rw-r--r-- 1 activemq java 1073741680 2009-01-26 19:56 data-1
-rw-r--r-- 1 activemq java 1073217536 2009-01-26 22:14 data-2
-rw-r--r-- 1 activemq java       2088 2009-01-26 22:14 data-control

Personally I don't trust the "hash-index-queue-data_queue" file in the
kr-store/data directory:

ls -l data/kr-store/data
-rw-r--r-- 1 activemq java         271 2009-01-26 18:35
data-container-roots-1
-rw-r--r-- 1 activemq java   104857602 2009-01-26 19:23 data-queue-data-2
-rw-r--r-- 1 activemq java   104857570 2009-01-26 20:01 data-queue-data-3
-rw-r--r-- 1 activemq java   104857570 2009-01-26 21:11 data-queue-data-4
-rw-r--r-- 1 activemq java    13787718 2009-01-26 21:25 data-queue-data-5
-rw-r--r-- 1 activemq java 17086873601 2009-01-26 22:18
hash-index-queue-data_queue#3a#2f#2ftracking
-rw-r--r-- 1 activemq java         153 2009-01-26 18:35
index-container-roots
-rw-r--r-- 1 activemq java   121943448 2009-01-26 22:18 index-queue-data
-rw-r--r-- 1 activemq java           0 2009-01-26 18:34 lock

How does it come that this file is respectable 16 GB big???
Could it be that maintaining the key does take a lot of time?
Is the Kahastore really faster than using a e.g. mysql DB?

I let my test continue until tomorrow, lets see if something new happens.

Thanks and best regards

J=C3=B6rg








Cybexion wrote:
>=20
> Hi,
>=20
> I have a queue that should hold up to 30 GB of small persistent messages:
>  <systemUsage sendFailIfNoSpace=3D"true">
>                 <memoryUsage>
>                     <memoryUsage limit=3D"1gb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit=3D"30gb" store=3D"#store"/>
>                 </storeUsage>
>=20
> The queue consists of 10 MB files and are specified like this:
>        <bean id=3D"store"
> class=3D"org.apache.activemq.store.amq.AMQPersistenceAdapter">
>                 <property name=3D"directory" value=3D"${activemq.base}/da=
ta"/>
>                 <property name=3D"maxFileLength" value=3D"10000000"/>
>                 <property name=3D"checkpointInterval" value=3D"60000"/>
>                 <property name=3D"cleanupInterval" value=3D"30000"/>
>                 <property name=3D"persistentIndex" value=3D"true"/>
>         </bean>
>=20
> This night I made a test with lots of produceds an just 5 consumer. I
> wanted to fill the queue to its limit.=20
> Well, so far I managed to send 18 Million messages into the queue. I have
> 20 % Store usage so far.=20
> What I noticed is, that mostly the messages are send within 0-20
> milliseconds. However sometimes I have a lot of messages which take 10+
> seconds to be send. This is not good in my scenario. I need a producer
> that should be very very fast in sending the messages.=20
> Question now:
> Could it be that the number of data files or the size of the datafiles ha=
s
> impact on the performance?
> Would it be better to have one queue big file with 30GB of size instead o=
f
> hundreds of files having 1 MB?
>=20
> The key index ist persistent. I tested this and it seems to be the only
> way of NOT getting an out of memory error.
> Does it make sense to change the indexkeysize or the indexpagesize in the
> store? Would this maybe gice additional constant performance?
>=20
> Btw:=20
> I'm using AMQ 5.2
>=20
> Thanks and best regards
>=20
> J=C3=B6rg
> =20
>=20
>=20
> =20
>=20
>=20
>=20

--=20
View this message in context: http://www.nabble.com/Performance-in-relation=
-to-queue-size-tp21601585p21673976.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.