activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: Producers hang under load with AMQ 5.2
Date Wed, 02 Sep 2009 12:36:47 GMT
Hi Tanel,

It would be worth disabling flow control - e.g.

disable flow control by setting the producerFlowControl flag to false  
on a destination policy in the Broker configuration - e.g.
<destinationPolicy>
       <policyMap>
         <policyEntries>

           <policyEntry topic="FOO.>" producerFlowControl="false"  
memoryLimit="1mb">
             <dispatchPolicy>
               <strictOrderDispatchPolicy/>
             </dispatchPolicy>
             <subscriptionRecoveryPolicy>
               <lastImageSubscriptionRecoveryPolicy/>
             </subscriptionRecoveryPolicy>
           </policyEntry>

         </policyEntries>
       </policyMap>
</destinationPolicy>
On 2 Sep 2009, at 12:35, wo wrote:

>
> Hi.
>
> We have 1 producer app and 2 consumers in total. Messages are  
> distributed
> evenly between 2 separate topics.
> Slow consumer should not be a problem, both consumers use
> java.util.concurrent Executors working off an unbounded queue to  
> process the
> incoming messages and every attempt has been made to make jms message
> handoff to executors as fast as possible.
>
> Messages itself are small, maybe 100 bytes of data in each. Our  
> topic memory
> limit in activemq config is set to 20mb which seems sensible enough.  
> Would
> it help to increase it further?
>
> I'll try to confirm whether the consumer is to blame but even if it  
> was, is
> there any workaround besides not putting a load on the system? :)  
> I'd say
> its a pretty major issue with activemq, one slow consumer shouldn't  
> be able
> to bring the whole thing down.
>
> Regards,
> Tanel
>
>
> Andreas Gies wrote:
>>
>> Hi there,
>>
>> I am wondering how many consumers there are on the topic and if there
>> are any slow consumers that
>> can't keep up with the messaging load. Without having verified it, it
>> seems that your app hangs in the
>> the send message waiting for the response from the broker. That would
>> happen, if the sending application
>> can't place the message in the broker.
>>
>> One thing strikes me odd....if all of your consumers keep consuming,
>> that shouldn't happen. So it might
>> be a starting point to double check them.
>>
>> As for the Advisory messages: Yes, thats normal. For more details on
>> advisories see
>> http://activemq.apache.org/advisory-message.html. You *could turn
>> advisories off,
>> but you need to remember to reenable them if you are switching to a
>> network of brokers
>> some day ;)
>>
>> Hope that helps a bit
>> Andreas
>>
>>
>> On Sep 2, 2009, at 7:24 AM, wo wrote:
>>
>>>
>>> Hi.
>>>
>>> We're having a problem under moderate to heavy load (~2500
>>> nonpersisted
>>> messages /second on a single broker) while producers will just hang
>>> indefinitely at some point and producer application needs to be
>>> restarted in
>>> order to continue.
>>>
>>> Here's a snippet from a producer application thread dump after hang:
>>>
>>> at
>>> java 
>>> .util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:
>>> 317)
>>>       at
>>> org
>>> .apache
>>> .activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>       at
>>> org
>>> .apache
>>> .activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:
>>> 1195)
>>>       at
>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1644)
>>>       - locked <0x00002aaab5abeb30> (a java.lang.Object)
>>>       at
>>> org
>>> .apache
>>> .activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:
>>> 227)
>>>       at
>>> org
>>> .apache
>>> .activemq
>>> .ActiveMQMessageProducerSupport
>>> .send(ActiveMQMessageProducerSupport.java:241)
>>>       at
>>> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java: 
>>> 597)
>>>
>>> So far we've tried upgrading AMQ from 5.1 to 5.2 and increasing the
>>> memory
>>> limits in broker config, both giving no result, the problem still
>>> exists.
>>> Neither application nor activemq log displays any errors /  
>>> exceptions.
>>>
>>> One thing i have noticed is that for every message sent to a topic,
>>> there
>>> are 2 advisory messages being produced in
>>> ActiveMQ.Advisory.Producer.Topic.TOPICNAME which strikes me as odd
>>> but i
>>> haven't been able to find any information in forums or manual about
>>> whether
>>> this is normal behaviour?
>>>
>>> Producer side is a standalone Spring application with AMQ
>>> connectionfactory
>>> wrapped in spring SingleConnectionFactory and used in jmstemplate,
>>> as per
>>> example.
>>>
>>> Any insight would be greatly appreciated.
>>>
>>> Tanel Unt
>>> Core Developer
>>> Tracking Center Ltd.
>>> -- 
>>> View this message in context:
>>> http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25251841.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25255812.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/





Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message