activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maarten_D <maarten.dir...@gmail.com>
Subject Re: 50k + messages stuck in queue with all consumers blocking on receive
Date Wed, 24 Feb 2010 17:27:35 GMT

Aha. And enqueued - dequeued corresponds to what's in the queue?


Elliot Barlas wrote:
> 
> Okay, the issue I am seeing is slightly different then.  In my case the
> broker isn't even dispatching some messages, at least according to the
> broker stats.  Dequeued = Dispatched and Dispatched < Enqueued.
> 
> Thanks,
> Elliot
> 
> 
> Maarten_D wrote:
>> 
>> It appears that our problem had to do with prefetching. We had our
>> prefetch values for queues set fairly high, and when I client application
>> would crash, a bunch of messages that were prefetched, but not ack'ed,
>> would remain stuck in the queue. We added
>> "jms.prefetchPolicy.queuePrefetch=1" to our connection uri, and haven't
>> seen this behaviour since.
>> 
>> 
>> Elliot Barlas wrote:
>>> 
>>> Hey Maarten, I am observing the same behavior in my AMQ application. 
>>> Was your problem resolved?  I've tried a few different connection
>>> factory approach to no avail :(
>>> 
>>> Thanks,
>>> Elliot
>>> 
>>> 
>>> 
>>> Maarten_D wrote:
>>>> 
>>>> Hi Rob,
>>>> Sorry, I'd killed that particular JVM before I read your request, and
>>>> the error hasn't reoccurred since then.
>>>> Something else that I've been thinking about since reading a 
>>>> http://tmielke.blogspot.com/2009/12/using-spring-jms-template-for-sending.html
>>>> blog post : we used the ActiveMQ PooledConnectionFactory for
>>>> establishing all connections to the broker, for the Spring message
>>>> listener containers as well as for the JMS templates.
>>>> After diving into the code of the PooledconnectionFactory, this seems
>>>> to have been a bad idea. When a connection is requested, the PCF simply
>>>> creates a new one and hands it out. When the pool is full, it returns
>>>> the first connection in its list (ie the first one it created), removes
>>>> it from the top of the list and adds it to the bottom. This means that,
>>>> if your listeners also get connections from this pool (and remember,
>>>> listeners hold on to their connections), a whole bunch of JMS template
>>>> calls will be made where the template is sending messages using the
>>>> same connector used by a listener.
>>>> I'm not too sure about the details, but when you introduce
>>>> producerflowcontrol into this picture, I can imagine how a kind of
>>>> deadlock can occur where eventually all producers on all connections
>>>> are throttled, leaving no one able to send any messages.
>>>> Does this sound like a plausible scenario?
>>>> 
>>>> We've modified our config and given all listener containers a
>>>> connection that's not in the pool, and are now running another test.
>>>> I'll post the results.
>>>> 
>>>> Regards,
>>>> Maarten
>>>> 
>>>> PS. Of course, you might say we should've know about this beforehand,
>>>> as the PCF javadoc says that its not really meant for consumers. The
>>>> alternative it offers is Jencks, although that project has been dead
>>>> for a while, to the point where even the homepage is now a spam site
>>>> (jencks.org). So clearly that isn't a viable alternative.
>>>> 
>>>> 
>>>> rajdavies wrote:
>>>>> 
>>>>> Can you take a thread dump whilst its in this state - and send us the
 
>>>>> output ?
>>>>> 
>>>>> thanks,
>>>>> 
>>>>> Rob
>>>>> On 21 Jan 2010, at 17:26, Maarten_D wrote:
>>>>> 
>>>>>>
>>>>>> O, and I forgot to mention I also turned on async sends
>>>>>> (jms.useAsyncSend=true)
>>>>>>
>>>>>> Maarten_D wrote:
>>>>>>>
>>>>>>> I've now changed my activemq.xml to the listing below, made the
 
>>>>>>> session
>>>>>>> transacted and set the acknowledge mode to SESSION_TRANSACTED.
>>>>>>>
>>>>>>> Things were going well for me for a while, with the system  
>>>>>>> processing 3,2
>>>>>>> million messages without a hitch, and then everything stopped
 
>>>>>>> because the
>>>>>>> first component in the chain got lots of these:
>>>>>>>
>>>>>>> javax.jms.InvalidClientIDException: Broker: broker - Client:
>>>>>>> ID:rhost-59116-1263927611185-1:445 already connected from / 
>>>>>>> 127.0.0.1:56560
>>>>>>>
>>>>>>> And for an hour now, since it stopped processing messages, the
 
>>>>>>> broker has
>>>>>>> been eating up almost 100% of the cpu for some reason I can't
quite  
>>>>>>> fathom
>>>>>>> (disk utilization is very low, and there is no message traffic
 
>>>>>>> passing
>>>>>>> through the broker).
>>>>>>>
>>>>>>> <beans  xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>>>>> http://activemq.apache.org/schema/core
>>>>>>> http://activemq.apache.org/schema/core/activemq-core.xsd
>>>>>>> http://mortbay.com/schemas/jetty/1.0
>>>>>>> http://jetty.mortbay.org/jetty.xsd 
>>>>>>> ">
>>>>>>>
>>>>>>>  <bean
>>>>>>> class 
>>>>>>> = 
>>>>>>> "org 
>>>>>>> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>>>>>    <property name="location" value="file:/etc/broker.properties"
/>
>>>>>>>  </bean>
>>>>>>>
>>>>>>>  <broker id="broker" useJmx="true" brokerName="${broker.name}"
>>>>>>> start="true" xmlns="http://activemq.apache.org/schema/core"
>>>>>>> dataDirectory="${activemq.data}">
>>>>>>>
>>>>>>>    <destinationPolicy>
>>>>>>>      <policyMap>
>>>>>>>        <policyEntries>
>>>>>>>          <policyEntry queue=">" memoryLimit="32mb"
>>>>>>> strictOrderDispatch="true" producerFlowControl="false">
>>>>>>>            <pendingQueuePolicy>
>>>>>>>            	<vmQueueCursor />
>>>>>>>            </pendingQueuePolicy>
>>>>>>>          </policyEntry>
>>>>>>>          <policyEntry topic=">" memoryLimit="32mb"
>>>>>>> producerFlowControl="true" />
>>>>>>>        </policyEntries>
>>>>>>>      </policyMap>
>>>>>>>    </destinationPolicy>
>>>>>>>
>>>>>>>    <managementContext>
>>>>>>>      <managementContext useMBeanServer="true"
>>>>>>>                         jmxDomainName="org.apache.activemq"
>>>>>>>                         createMBeanServer="true"
>>>>>>>                         createConnector="false"
>>>>>>>                         connectorPort="1100"
>>>>>>>                         connectorPath="/jmxrmi"/>
>>>>>>>    </managementContext>
>>>>>>>
>>>>>>>    <persistenceAdapter>
>>>>>>>      <kahaDB directory="${activemq.data}/${broker.name}"
>>>>>>>              journalMaxFileLength="32mb"
>>>>>>>              enableJournalDiskSyncs="false"
>>>>>>>              indexWriteBatchSize="1000"
>>>>>>>              indexCacheSize="1000"/>
>>>>>>>    </persistenceAdapter>
>>>>>>>
>>>>>>>    <systemUsage>
>>>>>>>      <systemUsage>
>>>>>>>        <memoryUsage>
>>>>>>>          <memoryUsage limit="512mb" />
>>>>>>>        </memoryUsage>
>>>>>>>      </systemUsage>
>>>>>>>    </systemUsage>
>>>>>>>
>>>>>>>    <transportConnectors>
>>>>>>>      <transportConnector uri="nio://0.0.0.0:61616" />
>>>>>>>    </transportConnectors>
>>>>>>>  </broker>
>>>>>>>
>>>>>>>  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>>>>>>>    <connectors>
>>>>>>>      <nioConnector port="61617"/>
>>>>>>>    </connectors>
>>>>>>>    <handlers>
>>>>>>>      <webAppContext contextPath="/admin"
>>>>>>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>>>>>>>    </handlers>
>>>>>>>  </jetty>
>>>>>>> </beans>
>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27261393.html
>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>
>>>>> 
>>>>> Rob Davies
>>>>> http://twitter.com/rajdavies
>>>>> I work here: http://fusesource.com
>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27714258.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message