activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: 50k + messages stuck in queue with all consumers blocking on receive
Date Mon, 18 Jan 2010 10:02:51 GMT
I think we'd really need to see the client code for the consumer too -  
its likely that the consumers are not acknowledging messages properly  
- so no more messages will be dispatched - as the broker thinks that  
all the consumers are still busy processing.

Things to be aware of when using Spring's JmsTemplate - http://activemq.apache.org/jmstemplate-gotchas.html

  and here http://activemq.apache.org/spring-support.html - but this  
article is extremely informative too - http://codedependents.com/2009/10/16/efficient-lightweight-jms-with-spring-and-activemq/

cheers,

Rob
On 18 Jan 2010, at 09:44, Maarten_D wrote:

>
> Anyone have any ideas?
>
>
> Maarten_D wrote:
>>
>> Hi,
>> I've somehow gotten myself into the situation where I've got 50,000+
>> messages stuck in a queue for no apparent reason. Allow me to  
>> explain:
>>
>> I've got a system where a component rapidly produces messages that  
>> are put
>> on topic (around 400 per second). A second component listens to the  
>> topic,
>> takes the information from the messages that it sees, repackages the
>> information in another message which it puts a on queue. A third  
>> component
>> eats from the queue, and processes the information in the messages.
>>
>> Under large load (upwards of 1.5m messages) we were experiencing  
>> system
>> hangs (no more messages were getting through), so I let loose the  
>> usual
>> stable of performance analysis tools (JConsole and Sar) to see what  
>> was
>> going one. Using the graphs produced with the sar info, you can  
>> clearly
>> see the points at which the producer flow control kicks in for the  
>> topic:
>> the cpu io-wait skyrockets and the JMX counters for the topic grind  
>> to a
>> halt. What's troubling, however, is that nothing else seems to be  
>> working
>> either (ie the downstream queues that were filled from the topic  
>> don't
>> seem to be emptied).
>>
>> Things got even stranger when I killed the producer, thereby  
>> cutting the
>> influx of new messages to the topic. The system is now in a stable  
>> state,
>> with the amount of enqueued messages on the topic equal to the  
>> dequeued
>> amount. However, there are more than 50,000 messages in the first  
>> queue,
>> waiting to be processed. All of the listeners that are supposed to  
>> eat
>> from this queue are blocked with the following stacktrace:
>>
>> java.lang.Thread.State: TIMED_WAITING on java.lang.Object@6e186c3f
>> 	at java.lang.Object.wait(Native Method)
>> 	at
>> org 
>> .apache 
>> .activemq 
>> .MessageDispatchChannel.dequeue(MessageDispatchChannel.java:77)
>> 	at
>> org 
>> .apache 
>> .activemq 
>> .ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:428)
>> 	at
>> org 
>> .apache 
>> .activemq 
>> .ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:554)
>> 	at
>> org 
>> .springframework 
>> .jms 
>> .listener 
>> .AbstractPollingMessageListenerContainer 
>> .receiveMessage(AbstractPollingMessageListenerContainer.java:405)
>> 	at
>> org 
>> .springframework 
>> .jms 
>> .listener 
>> .AbstractPollingMessageListenerContainer 
>> .doReceiveAndExecute(AbstractPollingMessageListenerContainer.java: 
>> 308)
>> 	at
>> org 
>> .springframework 
>> .jms 
>> .listener 
>> .AbstractPollingMessageListenerContainer 
>> .receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
>> 	at
>> org.springframework.jms.listener.DefaultMessageListenerContainer 
>> $ 
>> AsyncMessageListenerInvoker 
>> .invokeListener(DefaultMessageListenerContainer.java:982)
>> 	at
>> org.springframework.jms.listener.DefaultMessageListenerContainer 
>> $ 
>> AsyncMessageListenerInvoker 
>> .executeOngoingLoop(DefaultMessageListenerContainer.java:974)
>> 	at
>> org.springframework.jms.listener.DefaultMessageListenerContainer 
>> $ 
>> AsyncMessageListenerInvoker 
>> .run(DefaultMessageListenerContainer.java:876)
>> 	at java.lang.Thread.run(Thread.java:619)
>>
>> I've used JConsole to stop and start the (tcp) connector several  
>> times,
>> and each time (strangely) around 2075 messages have been eaten from  
>> the
>> queue by the consumers, after which things freeze again. If I  
>> restart the
>> entire broker, around 800 messages are eaten from the queue before  
>> things
>> stagnate again.
>>
>> My basic question is: what is going on, and how can I prevent those
>> messages from getting stuck in the queue?
>>
>> To make matters even more interesting, I ran another test a while ago
>> where 10 million messages were fed to the same setup without a  
>> hitch. The
>> only difference between that test and the one I'm running now was the
>> nature of the message. The average size of a message in both sets is
>> almost equal, but during the current test message sizes vary more  
>> than
>> during the succesfull one.
>>
>> Anyone have any ideas? Below are a bunch of relevant settings an my
>> activemq.xml config. We code to activemq using the Spring  
>> JMSTemplate.
>>
>> Thanks in advance,
>> Maarten
>>
>> ActiveMQ: 5.3.0
>> Java: 1.6.0_17
>> Spring: 2.5.6
>> Connector URL: tcp://localhost:61616
>> JMS receivetimeout: 30000
>> JMS Acknowledgemode: CLIENT_ACKNOWLEDGE
>> JMS Session transacted: false
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:amq="http://activemq.apache.org/schema/core"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core
>> http://activemq.apache.org/schema/core/activemq-core.xsd
>> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd 
>> ">
>>
>>  <broker id="broker" useJmx="true" brokerName="testbroker"  
>> start="true"
>> xmlns="http://activemq.apache.org/schema/core"
>> dataDirectory="/var/amqdata">
>>
>>    <destinationPolicy>
>>      <policyMap>
>>        <policyEntries>
>>          <policyEntry queue=">" memoryLimit="32 mb"
>> producerFlowControl="true" />
>>          <policyEntry topic=">" memoryLimit="32 mb"
>> producerFlowControl="true" />
>>        </policyEntries>
>>      </policyMap>
>>    </destinationPolicy>
>>
>>   <managementContext>
>>     <managementContext
>>       useMBeanServer="true"
>>       jmxDomainName="org.apache.activemq"
>>       createMBeanServer="true"
>>       createConnector="false"
>>       connectorPort="1100"
>>       connectorPath="/jmxrmi"/>
>>  </managementContext>
>>
>>  <persistenceAdapter>
>>    <amqPersistenceAdapter
>>      syncOnWrite="false"
>>      directory="/var/amqdata/testbroker"
>>      indexBinSize="8192"
>>      cleanupInterval="300000"
>>      indexPageSize="64 kb"
>>      maxFileLength="256 mb"
>>      archiveDataLogs="false"/>
>>    </persistenceAdapter>
>>
>>    <systemUsage>
>>      <systemUsage>
>>        <memoryUsage>
>>          <memoryUsage limit="512 mb" />
>>        </memoryUsage>
>>      </systemUsage>
>>    </systemUsage>
>>
>>    <transportConnectors>
>>      <transportConnector uri="tcp://localhost:61616" />
>>    </transportConnectors>
>>
>>  </broker>
>>
>> </beans>
>>
>
> -- 
> View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27208079.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message