Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates
 216.139.236.158 as permitted sender)
Message-ID: <27664172.post@talk.nabble.com>
Date: Fri, 19 Feb 2010 23:21:28 -0800 (PST)
From: Elliot Barlas <elliotbarlas@gmail.com>
To: users@activemq.apache.org
Subject: Re: 50k + messages stuck in queue with all consumers blocking on
 receive
In-Reply-To: <27592645.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: <27162095.post@talk.nabble.com> <27208079.post@talk.nabble.com>
 <B9932902-795D-4028-B510-55B41A44F5BB@gmail.com>
 <27210211.post@talk.nabble.com>
 <30B59B0B-8228-4A7A-98D7-E21219F1E0AD@pobox.com>
 <122EB55A-2EB3-4BDF-88BD-3C3FE1177441@gmail.com>
 <27261165.post@talk.nabble.com> <27261393.post@talk.nabble.com>
 <8D28B201-CD25-4DC6-B158-ED73292945F2@gmail.com>
 <27592645.post@talk.nabble.com>


Hey Maarten, I am observing the same behavior in my AMQ application.  Was
your problem resolved?  I've tried a few different connection factory
approach to no avail :(

Thanks,
Elliot


Maarten_D wrote:
> 
> Hi Rob,
> Sorry, I'd killed that particular JVM before I read your request, and the
> error hasn't reoccurred since then.
> Something else that I've been thinking about since reading a 
> http://tmielke.blogspot.com/2009/12/using-spring-jms-template-for-sending.html
> blog post : we used the ActiveMQ PooledConnectionFactory for establishing
> all connections to the broker, for the Spring message listener containers
> as well as for the JMS templates.
> After diving into the code of the PooledconnectionFactory, this seems to
> have been a bad idea. When a connection is requested, the PCF simply
> creates a new one and hands it out. When the pool is full, it returns the
> first connection in its list (ie the first one it created), removes it
> from the top of the list and adds it to the bottom. This means that, if
> your listeners also get connections from this pool (and remember,
> listeners hold on to their connections), a whole bunch of JMS template
> calls will be made where the template is sending messages using the same
> connector used by a listener.
> I'm not too sure about the details, but when you introduce
> producerflowcontrol into this picture, I can imagine how a kind of
> deadlock can occur where eventually all producers on all connections are
> throttled, leaving no one able to send any messages.
> Does this sound like a plausible scenario?
> 
> We've modified our config and given all listener containers a connection
> that's not in the pool, and are now running another test. I'll post the
> results.
> 
> Regards,
> Maarten
> 
> PS. Of course, you might say we should've know about this beforehand, as
> the PCF javadoc says that its not really meant for consumers. The
> alternative it offers is Jencks, although that project has been dead for a
> while, to the point where even the homepage is now a spam site
> (jencks.org). So clearly that isn't a viable alternative.
> 
> 
> rajdavies wrote:
>> 
>> Can you take a thread dump whilst its in this state - and send us the  
>> output ?
>> 
>> thanks,
>> 
>> Rob
>> On 21 Jan 2010, at 17:26, Maarten_D wrote:
>> 
>>>
>>> O, and I forgot to mention I also turned on async sends
>>> (jms.useAsyncSend=true)
>>>
>>> Maarten_D wrote:
>>>>
>>>> I've now changed my activemq.xml to the listing below, made the  
>>>> session
>>>> transacted and set the acknowledge mode to SESSION_TRANSACTED.
>>>>
>>>> Things were going well for me for a while, with the system  
>>>> processing 3,2
>>>> million messages without a hitch, and then everything stopped  
>>>> because the
>>>> first component in the chain got lots of these:
>>>>
>>>> javax.jms.InvalidClientIDException: Broker: broker - Client:
>>>> ID:rhost-59116-1263927611185-1:445 already connected from / 
>>>> 127.0.0.1:56560
>>>>
>>>> And for an hour now, since it stopped processing messages, the  
>>>> broker has
>>>> been eating up almost 100% of the cpu for some reason I can't quite  
>>>> fathom
>>>> (disk utilization is very low, and there is no message traffic  
>>>> passing
>>>> through the broker).
>>>>
>>>> <beans  xmlns="http://www.springframework.org/schema/beans"
>>>> xmlns:amq="http://activemq.apache.org/schema/core"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>>> http://activemq.apache.org/schema/core
>>>> http://activemq.apache.org/schema/core/activemq-core.xsd
>>>> http://mortbay.com/schemas/jetty/1.0 http://jetty.mortbay.org/jetty.xsd 
>>>> ">
>>>>
>>>>  <bean
>>>> class 
>>>> = 
>>>> "org 
>>>> .springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>>>>    <property name="location" value="file:/etc/broker.properties" />
>>>>  </bean>
>>>>
>>>>  <broker id="broker" useJmx="true" brokerName="${broker.name}"
>>>> start="true" xmlns="http://activemq.apache.org/schema/core"
>>>> dataDirectory="${activemq.data}">
>>>>
>>>>    <destinationPolicy>
>>>>      <policyMap>
>>>>        <policyEntries>
>>>>          <policyEntry queue=">" memoryLimit="32mb"
>>>> strictOrderDispatch="true" producerFlowControl="false">
>>>>            <pendingQueuePolicy>
>>>>            	<vmQueueCursor />
>>>>            </pendingQueuePolicy>
>>>>          </policyEntry>
>>>>          <policyEntry topic=">" memoryLimit="32mb"
>>>> producerFlowControl="true" />
>>>>        </policyEntries>
>>>>      </policyMap>
>>>>    </destinationPolicy>
>>>>
>>>>    <managementContext>
>>>>      <managementContext useMBeanServer="true"
>>>>                         jmxDomainName="org.apache.activemq"
>>>>                         createMBeanServer="true"
>>>>                         createConnector="false"
>>>>                         connectorPort="1100"
>>>>                         connectorPath="/jmxrmi"/>
>>>>    </managementContext>
>>>>
>>>>    <persistenceAdapter>
>>>>      <kahaDB directory="${activemq.data}/${broker.name}"
>>>>              journalMaxFileLength="32mb"
>>>>              enableJournalDiskSyncs="false"
>>>>              indexWriteBatchSize="1000"
>>>>              indexCacheSize="1000"/>
>>>>    </persistenceAdapter>
>>>>
>>>>    <systemUsage>
>>>>      <systemUsage>
>>>>        <memoryUsage>
>>>>          <memoryUsage limit="512mb" />
>>>>        </memoryUsage>
>>>>      </systemUsage>
>>>>    </systemUsage>
>>>>
>>>>    <transportConnectors>
>>>>      <transportConnector uri="nio://0.0.0.0:61616" />
>>>>    </transportConnectors>
>>>>  </broker>
>>>>
>>>>  <jetty xmlns="http://mortbay.com/schemas/jetty/1.0">
>>>>    <connectors>
>>>>      <nioConnector port="61617"/>
>>>>    </connectors>
>>>>    <handlers>
>>>>      <webAppContext contextPath="/admin"
>>>> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/>
>>>>    </handlers>
>>>>  </jetty>
>>>> </beans>
>>>>
>>>
>>> -- 
>>> View this message in context:
>>> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27261393.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>> 
>> Rob Davies
>> http://twitter.com/rajdavies
>> I work here: http://fusesource.com
>> My Blog: http://rajdavies.blogspot.com/
>> I'm writing this: http://www.manning.com/snyder/
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-tp27162095p27664172.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.