activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Gies <andr...@soa-knowledge.net>
Subject Re: Producers hang under load with AMQ 5.2
Date Wed, 02 Sep 2009 19:38:38 GMT
Hi again,

if it turns out to be a consumer, you could use the concept of a  
Virtual topics to have several groups of consumers,
each group getting a message copy, but the group members sharing the  
load.

To your question, You need to decide on a strategy what you do in case  
of slow consumers. I think you are experiencing
the default - flow control. See Rob's suggestion as well. Besides you  
*could* configure the broker in a way that the send throws an exception
if the situation occurs, but then that exception must be handled on  
the producer site.


Best regards
Andreas


On Sep 2, 2009, at 2:36 PM, Rob Davies wrote:

> Hi Tanel,
>
> It would be worth disabling flow control - e.g.
>
> disable flow control by setting the producerFlowControl flag to  
> false on a destination policy in the Broker configuration - e.g.
> <destinationPolicy>
>      <policyMap>
>        <policyEntries>
>
>          <policyEntry topic="FOO.>" producerFlowControl="false"  
> memoryLimit="1mb">
>            <dispatchPolicy>
>              <strictOrderDispatchPolicy/>
>            </dispatchPolicy>
>            <subscriptionRecoveryPolicy>
>              <lastImageSubscriptionRecoveryPolicy/>
>            </subscriptionRecoveryPolicy>
>          </policyEntry>
>
>        </policyEntries>
>      </policyMap>
> </destinationPolicy>
> On 2 Sep 2009, at 12:35, wo wrote:
>
>>
>> Hi.
>>
>> We have 1 producer app and 2 consumers in total. Messages are  
>> distributed
>> evenly between 2 separate topics.
>> Slow consumer should not be a problem, both consumers use
>> java.util.concurrent Executors working off an unbounded queue to  
>> process the
>> incoming messages and every attempt has been made to make jms message
>> handoff to executors as fast as possible.
>>
>> Messages itself are small, maybe 100 bytes of data in each. Our  
>> topic memory
>> limit in activemq config is set to 20mb which seems sensible  
>> enough. Would
>> it help to increase it further?
>>
>> I'll try to confirm whether the consumer is to blame but even if it  
>> was, is
>> there any workaround besides not putting a load on the system? :)  
>> I'd say
>> its a pretty major issue with activemq, one slow consumer shouldn't  
>> be able
>> to bring the whole thing down.
>>
>> Regards,
>> Tanel
>>
>>
>> Andreas Gies wrote:
>>>
>>> Hi there,
>>>
>>> I am wondering how many consumers there are on the topic and if  
>>> there
>>> are any slow consumers that
>>> can't keep up with the messaging load. Without having verified it,  
>>> it
>>> seems that your app hangs in the
>>> the send message waiting for the response from the broker. That  
>>> would
>>> happen, if the sending application
>>> can't place the message in the broker.
>>>
>>> One thing strikes me odd....if all of your consumers keep consuming,
>>> that shouldn't happen. So it might
>>> be a starting point to double check them.
>>>
>>> As for the Advisory messages: Yes, thats normal. For more details on
>>> advisories see
>>> http://activemq.apache.org/advisory-message.html. You *could turn
>>> advisories off,
>>> but you need to remember to reenable them if you are switching to a
>>> network of brokers
>>> some day ;)
>>>
>>> Hope that helps a bit
>>> Andreas
>>>
>>>
>>> On Sep 2, 2009, at 7:24 AM, wo wrote:
>>>
>>>>
>>>> Hi.
>>>>
>>>> We're having a problem under moderate to heavy load (~2500
>>>> nonpersisted
>>>> messages /second on a single broker) while producers will just hang
>>>> indefinitely at some point and producer application needs to be
>>>> restarted in
>>>> order to continue.
>>>>
>>>> Here's a snippet from a producer application thread dump after  
>>>> hang:
>>>>
>>>> at
>>>> java 
>>>> .util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:
>>>> 317)
>>>>      at
>>>> org
>>>> .apache
>>>> .activemq.transport.FutureResponse.getResult(FutureResponse.java: 
>>>> 40)
>>>>      at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>>      at
>>>> org
>>>> .apache
>>>> .activemq 
>>>> .ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:
>>>> 1195)
>>>>      at
>>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1644)
>>>>      - locked <0x00002aaab5abeb30> (a java.lang.Object)
>>>>      at
>>>> org
>>>> .apache
>>>> .activemq 
>>>> .ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:
>>>> 227)
>>>>      at
>>>> org
>>>> .apache
>>>> .activemq
>>>> .ActiveMQMessageProducerSupport
>>>> .send(ActiveMQMessageProducerSupport.java:241)
>>>>      at
>>>> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java: 
>>>> 597)
>>>>
>>>> So far we've tried upgrading AMQ from 5.1 to 5.2 and increasing the
>>>> memory
>>>> limits in broker config, both giving no result, the problem still
>>>> exists.
>>>> Neither application nor activemq log displays any errors /  
>>>> exceptions.
>>>>
>>>> One thing i have noticed is that for every message sent to a topic,
>>>> there
>>>> are 2 advisory messages being produced in
>>>> ActiveMQ.Advisory.Producer.Topic.TOPICNAME which strikes me as odd
>>>> but i
>>>> haven't been able to find any information in forums or manual about
>>>> whether
>>>> this is normal behaviour?
>>>>
>>>> Producer side is a standalone Spring application with AMQ
>>>> connectionfactory
>>>> wrapped in spring SingleConnectionFactory and used in jmstemplate,
>>>> as per
>>>> example.
>>>>
>>>> Any insight would be greatly appreciated.
>>>>
>>>> Tanel Unt
>>>> Core Developer
>>>> Tracking Center Ltd.
>>>> -- 
>>>> View this message in context:
>>>> http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25251841.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>>
>>
>> -- 
>> View this message in context: http://www.nabble.com/Producers-hang-under-load-with-AMQ-5.2-tp25251841p25255812.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>


Mime
View raw message