Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted
 nor denied by domain of harishs@yahoo-inc.com)
From: Harish Sharma <harishs@yahoo-inc.com>
To: "users@activemq.apache.org" <users@activemq.apache.org>
CC: "torsten@fusesource.com" <torsten@fusesource.com>
Date: Wed, 21 Dec 2011 15:21:21 +0530
Subject: Re: Backlog data causes producer to slow down.
Thread-Topic: Backlog data causes producer to slow down.
Thread-Index: Acy/xhgbdAT19GvmSLWnCdqZPPmsDA==
Message-ID: <C182A078-A4D5-4757-B7F4-97528D1689A1@yahoo-inc.com>
References: <924C8723-0E88-43C0-82CD-2870DDC7E90B@yahoo-inc.com>
 <9C3D04DB-9989-4134-8DB1-6B2CD369D763@fusesource.com>
In-Reply-To: <9C3D04DB-9989-4134-8DB1-6B2CD369D763@fusesource.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Torsten,

yea i tried with optimizedDispatch=3D"false" also.
And i am using queue.=20
Here is my kahaDb settings :
<persistenceAdapter>
      <kahaDB directory=3D"data/kahadb" concurrentStoreAndDispatchQueues=3D=
"false" enableIndexWriteAsync=3D"true" enableJournalDiskSyncs=3D"false" max=
AsyncJobs=3D"50000" cleanupInterval=3D"60000" checkpointInterval=3D"120000"=
 journalMaxFileLength=3D"1g" indexCacheSize=3D"300000" indexWriteBatchSize=
=3D"2147483647" /> =20
    </persistenceAdapter>

And i also tried with enableIndexWriteAsync=3D"false"  but nothing changes =
much.
Also when i ran profiler then i found out when the consumer starts for the =
first time the producer thread goes in to waiting state for the lock on the=
 AbstractQueuedSynchronizer ( the queue it is supposed to write in).
Actually the problem is the degradation in message production rate . I can =
work with low production rate but what is happening is due to consumers reg=
ularly going up and down the production rate actually goes to near Zero val=
ue.

Right now i am using Vmconnector for both producer and consumer.Do you have=
 any suggestion here.What is the best choice .( I am thinking of using NIO =
for producers).

T&R
Harish Sharma

On Dec 21, 2011, at 2:43 PM, Torsten Mielke wrote:

> Hard to say what exactly the problem is. In the past I got to learn the o=
ptimizedDispatch=3D"true" isn't always a good choice. Can you try without i=
t?
>=20
> In addition I recently found that enableIndexWriteAsync=3D"true" on your =
kahadb config may not bring any performance improvement. Not sure if you ha=
ve set this.
>=20
> Also does your test involve queue or topic messages?
>=20
>=20
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>=20
>=20
>=20
> On Dec 20, 2011, at 10:37 AM, Harish Sharma wrote:
>=20
>> Hi,
>> I tried to reply to http://activemq.2283324.n4.nabble.com/Backlog-data-c=
auses-producers-to-slow-down-tt3806018.html post . Because i am facing simi=
lar issue.
>> Whenever my consumer goes down and there is a data back log the producti=
on rate goes down significantly.
>> Moreover when my consumer is up again , the production rate goes down ag=
ain and this cycle repeats itself.
>> Some Info :
>> I am using ActiveMQ 5.5.1 , KahaDB message store,  Producer flow control=
 =3D FALSE , AsyncSend(true) , persistence =3D true and ConcurrentQueueStor=
eAndDispatch =3D true and false (tried both does not make much difference)
>> As i read in this post about cursors so i am using the default store cur=
sor.
>>=20
>> Here is snippet from broker.xml
>> destinationPolicy>
>>     <policyMap>
>>       <policyEntries>
>>         <policyEntry topic=3D">" optimizedDispatch=3D"true" memoryLimit=
=3D"1gb" producerFlowControl=3D"false"  />
>>         <policyEntry queue=3D">" optimizedDispatch=3D"true" memoryLimit=
=3D"1gb" producerFlowControl=3D"false" />
>>       </policyEntries>
>>     </policyMap>
>>   </destinationPolicy>
>>=20
>> <systemUsage>
>>     <systemUsage sendFailIfNoSpace=3D"true">
>>       <memoryUsage>
>>         <memoryUsage limit=3D"20 gb" />
>>       </memoryUsage>
>>       <storeUsage>
>>         <storeUsage limit=3D"500 gb" name=3D"foo" />
>>       </storeUsage>
>>       <tempUsage>
>>         <tempUsage limit=3D"1 gb" />
>>       </tempUsage>
>>     </systemUsage>
>>   </systemUsage>
>>=20
>> I am using all the optimizations(i read from http://fusesource.com/docs/=
broker/5.4/tuning/index.html )
>> for producer , consumer and broker.
>> These optimizations are able to increase message production rate a bit b=
ut the main issue is the stability of the production rate.(why it goes down=
 10 times and then keep on going down with every time consumers goes down o=
r goes up again).
>> I am trying to figure out the cause but unable to pinpoint the code in a=
ctiveMQ which is causing this issue.
>> When i saw threads state in jconsole i found out that as soon as consume=
r start running , producer thread goes into wait state ( State: WAITING on =
java.util.concurrent.locks.AbstractQueuedSynchronizer ).I understand the qu=
eue these producers write to is blocking queue.But i  don't understand why =
consumers  are affecting producers while flow control is off and my store s=
ize is 500 GB and memory is 20GB .Why not producers are producing at a cons=
tant rate until my KahaDB store is full.
>>=20
>> I really need help in this matter and deeply appreciate any help from yo=
u guys.
>>=20
>> Thanks & Regards
>> Harish Sharma
>=20
>=20
>=20
>=20

T&R
Harish Sharma