activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maarten_D <maarten.dir...@gmail.com>
Subject Re: 50k + messages stuck in queue with all consumers blocking on receive
Date Mon, 15 Feb 2010 11:41:58 GMT

Hi Rob,
Sorry, I'd killed that particular JVM before I read your request, and the
error hasn't reoccurred since then.
Something else that I've been thinking about since reading a 
http://tmielke.blogspot.com/2009/12/using-spring-jms-template-for-sending.html
blog post : we used the ActiveMQ PooledConnectionFactory for establishing
all connections to the broker, for the Spring message listener containers as
well as for the JMS templates.
After diving into the code of the PooledconnectionFactory, this seems to
have been a bad idea. When a connection is requested, the PCF simply creates
a new one and hands it out. When the pool is full, it returns the first
connection in its list (ie the first one it created), removes it from the
top of the list and adds it to the bottom. This means that, if your
listeners also get connections from this pool (and remember, listeners hold
on to their connections), a whole bunch of JMS template calls will be made
where the template is sending messages using the same connector used by a
listener.
I'm not too sure about the details, but when you introduce
producerflowcontrol into this picture, I can imagine how a kind of deadlock
can occur where eventually all producers on all connections are throttled,
leaving no one able to send any messages.
Does this sound like a plausible scenario?

We've modified our config and given all listener containers a connection
that's not in the pool, and are now running another test. I'll post the
results.

Regards,
Maarten

PS. Of course, you might say we should've know about this beforehand, as the
PCF javadoc says that its not really meant for consumers. The alternative it
offers is Jencks, although that project has been dead for a while, to the
point where even the homepage is now a spam site (jencks.org). So clearly
that isn't a viable alternative.


rajdavies wrote:
> 
> Can you take a thread dump whilst its in this state - and send us the  
> output ?
> 
> thanks,
> 
> Rob
> On 21 Jan 2010, at 17:26, Maarten_D wrote:
> 
>>
>> O, and I forgot to mention I also turned on async sends
>> (jms.useAsyncSend=true)
>>
>> Maarten_D wrote:
>>>
>>> I've now changed my activemq.xml to the listing below, made the  
>>> session
>>> transacted and set the acknowledge mode to SESSION_TRANSACTED.
>>>
>>> Things were going well for me for a while, with the system  
>>> processing 3,2
>>> million messages without a hitch, and then everything stopped  
>>> because the
>>> first component in the chain got lots of these:
>>>
>>> javax.jms.InvalidClientIDException: Broker: broker - Client:
>>> ID:rhost-59116-1263927611185-1:445 already connected from / 
>>> 127.0.0.1:56560
>>>
>>> And for an hour now, since it stopped processing messages, the  
>>> broker has
>>> been eating up almost 100% of the cpu for some reason I can't quite  
>>> fathom
>>> (disk utilization is very low, and there is no message traffic  
>>> passing
>>> through the broker).
>>>
>>> <beans  xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>> http://activemq.apache.org/schema/core
>>> http://activemq.apache.org/schema/core/activemq-core.xsd
>>> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd 
>>> ">
>>>
>>>  <bean
>>> class 
>>> = 
>>> "org 
>>> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>    <property name="location" value="file:/etc/broker.properties" />
>>>  </bean>
>>>
>>>  <broker id="broker" useJmx="true" brokerName="${broker.name}"
>>> start="true" xmlns="http://activemq.apache.org/schema/core"
>>> dataDirectory="${activemq.data}">
>>>
>>>    <destinationPolicy>
>>>      <policyMap>
>>>        <policyEntries>
>>>          <policyEntry queue=">" memoryLimit="32mb"
>>> strictOrderDispatch="true" producerFlowControl="false">
>>>            <pendingQueuePolicy>
>>>            	<vmQueueCursor />
>>>            </pendingQueuePolicy>
>>>          </policyEntry>
>>>          <policyEntry topic=">" memoryLimit="32mb"
>>> producerFlowControl="true" />
>>>        </policyEntries>
>>>      </policyMap>
>>>    </destinationPolicy>
>>>
>>>    <managementContext>
>>>      <managementContext useMBeanServer="true"
>>>                         jmxDomainName="org.apache.activemq"
>>>                         createMBeanServer="true"
>>>                         createConnector="false"
>>>                         connectorPort="1100"
>>>                         connectorPath="/jmxrmi"/>
>>>    </managementContext>
>>>
>>>    <persistenceAdapter>
>>>      <kahaDB directory="${activemq.data}/${broker.name}"
>>>              journalMaxFileLength="32mb"
>>>              enableJournalDiskSyncs="false"
>>>              indexWriteBatchSize="1000"
>>>              indexCacheSize="1000"/>
>>>    </persistenceAdapter>
>>>
>>>    <systemUsage>
>>>      <systemUsage>
>>>        <memoryUsage>
>>>          <memoryUsage limit="512mb" />
>>>        </memoryUsage>
>>>      </systemUsage>
>>>    </systemUsage>
>>>
>>>    <transportConnectors>
>>>      <transportConnector uri="nio://0.0.0.0:61616" />
>>>    </transportConnectors>
>>>  </broker>
>>>
>>>  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>>>    <connectors>
>>>      <nioConnector port="61617"/>
>>>    </connectors>
>>>    <handlers>
>>>      <webAppContext contextPath="/admin"
>>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>>>    </handlers>
>>>  </jetty>
>>> </beans>
>>>
>>
>> -- 
>> View this message in context:
>> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27261393.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27592645.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message