activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish Sharma <hari...@yahoo-inc.com>
Subject Re: Backlog data causes producer to slow down.
Date Wed, 21 Dec 2011 09:51:21 GMT
Hi Torsten,

yea i tried with optimizedDispatch="false" also.
And i am using queue. 
Here is my kahaDb settings :
<persistenceAdapter>
      <kahaDB directory="data/kahadb" concurrentStoreAndDispatchQueues="false" enableIndexWriteAsync="true"
enableJournalDiskSyncs="false" maxAsyncJobs="50000" cleanupInterval="60000" checkpointInterval="120000"
journalMaxFileLength="1g" indexCacheSize="300000" indexWriteBatchSize="2147483647" /> 

    </persistenceAdapter>

And i also tried with enableIndexWriteAsync="false"  but nothing changes much.
Also when i ran profiler then i found out when the consumer starts for the first time the
producer thread goes in to waiting state for the lock on the AbstractQueuedSynchronizer (
the queue it is supposed to write in).
Actually the problem is the degradation in message production rate . I can work with low production
rate but what is happening is due to consumers regularly going up and down the production
rate actually goes to near Zero value.

Right now i am using Vmconnector for both producer and consumer.Do you have any suggestion
here.What is the best choice .( I am thinking of using NIO for producers).

T&R
Harish Sharma

On Dec 21, 2011, at 2:43 PM, Torsten Mielke wrote:

> Hard to say what exactly the problem is. In the past I got to learn the optimizedDispatch="true"
isn't always a good choice. Can you try without it?
> 
> In addition I recently found that enableIndexWriteAsync="true" on your kahadb config
may not bring any performance improvement. Not sure if you have set this.
> 
> Also does your test involve queue or topic messages?
> 
> 
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
> 
> 
> 
> On Dec 20, 2011, at 10:37 AM, Harish Sharma wrote:
> 
>> Hi,
>> I tried to reply to http://activemq.2283324.n4.nabble.com/Backlog-data-causes-producers-to-slow-down-tt3806018.html
post . Because i am facing similar issue.
>> Whenever my consumer goes down and there is a data back log the production rate goes
down significantly.
>> Moreover when my consumer is up again , the production rate goes down again and this
cycle repeats itself.
>> Some Info :
>> I am using ActiveMQ 5.5.1 , KahaDB message store,  Producer flow control = FALSE
, AsyncSend(true) , persistence = true and ConcurrentQueueStoreAndDispatch = true and false
(tried both does not make much difference)
>> As i read in this post about cursors so i am using the default store cursor.
>> 
>> Here is snippet from broker.xml
>> destinationPolicy>
>>     <policyMap>
>>       <policyEntries>
>>         <policyEntry topic=">" optimizedDispatch="true" memoryLimit="1gb" producerFlowControl="false"
 />
>>         <policyEntry queue=">" optimizedDispatch="true" memoryLimit="1gb" producerFlowControl="false"
/>
>>       </policyEntries>
>>     </policyMap>
>>   </destinationPolicy>
>> 
>> <systemUsage>
>>     <systemUsage sendFailIfNoSpace="true">
>>       <memoryUsage>
>>         <memoryUsage limit="20 gb" />
>>       </memoryUsage>
>>       <storeUsage>
>>         <storeUsage limit="500 gb" name="foo" />
>>       </storeUsage>
>>       <tempUsage>
>>         <tempUsage limit="1 gb" />
>>       </tempUsage>
>>     </systemUsage>
>>   </systemUsage>
>> 
>> I am using all the optimizations(i read from http://fusesource.com/docs/broker/5.4/tuning/index.html
)
>> for producer , consumer and broker.
>> These optimizations are able to increase message production rate a bit but the main
issue is the stability of the production rate.(why it goes down 10 times and then keep on
going down with every time consumers goes down or goes up again).
>> I am trying to figure out the cause but unable to pinpoint the code in activeMQ which
is causing this issue.
>> When i saw threads state in jconsole i found out that as soon as consumer start running
, producer thread goes into wait state ( State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer
).I understand the queue these producers write to is blocking queue.But i  don't understand
why consumers  are affecting producers while flow control is off and my store size is 500
GB and memory is 20GB .Why not producers are producing at a constant rate until my KahaDB
store is full.
>> 
>> I really need help in this matter and deeply appreciate any help from you guys.
>> 
>> Thanks & Regards
>> Harish Sharma
> 
> 
> 
> 

T&R
Harish Sharma




Mime
View raw message