activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: consumer threads waiting on VMTransport locks...
Date Wed, 24 Jul 2013 04:19:01 GMT
Any chance you can reproduce this?


On Tue, Jul 23, 2013 at 1:03 AM, boday <ben.oday@initekconsulting.com>wrote:

> no errors in the logs, but we see message flow resume after a server
> restart...I'm starting to question our connection/server configuration a
> bit
> now given this thread dump...
>
> our application uses camel-activemq (2.10.1) and runs embedded in an AMQ
> broker (5.7)....use the same connection factory/pool for all of our apps
> camel routes that produce/consume from queues, etc...
>
> here is my client code connection setup...
>
>         ActiveMQConnectionFactory queueConnectionFactory = new
> ActiveMQConnectionFactory("vm://localhost?create=false");
>     queueConnectionFactory.setObjectMessageSerializationDefered(true);
>         queueConnectionFactory.setOptimizedMessageDispatch(true);
>         queueConnectionFactory.setDispatchAsync(true);
>         queueConnectionFactory.setUseAsyncSend(true);
>         queueConnectionFactory.setAlwaysSessionAsync(true);
>
>     PooledConnectionFactory pcf = new
> PooledConnectionFactory(queueConnectionFactory);
>         pcf.setIdleTimeout(0);
>
>         ActiveMQComponent amq =
> ActiveMQComponent.activeMQComponent(brokerURL);
>         amq.setConnectionFactory(pcf);
>         amq.setMaxConcurrentConsumers(20);
>         amq.setCacheLevelName("CACHE_CONSUMER");
>         amq.setAcknowledgementModeName("AUTO_ACKNOWLEDGE");
>         this.getContext().addComponent("activemq", amq);
>
> here is my activemq.xml...
>
>    <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="../data" useJmx="true">
>
>         <destinationPolicy>
>                <policyMap>
>                   <policyEntries>
>                    <policyEntry queue=">" producerFlowControl="false">
>                    <pendingQueuePolicy>
>                      <vmQueueCursor/>
>                    </pendingQueuePolicy>
>                   </policyEntry>
>                   </policyEntries>
>                </policyMap>
>         </destinationPolicy>
>
>         <managementContext>
>             <managementContext createConnector="false"/>
>         </managementContext>
>
>         <networkConnectors>
>         </networkConnectors>
>
>         <persistenceAdapter>
>             <kahaDB directory="../data/kaha"
>                 enableJournalDiskSyncs="false"
>                 indexWriteBatchSize="10000"
>                 indexCacheSize="1000" />
>         </persistenceAdapter>
>
>         <systemUsage>
>             <systemUsage>
>                 <memoryUsage><memoryUsage limit="4 gb"/></memoryUsage>
>                 <storeUsage><storeUsage limit="100 gb"/></storeUsage>
>                 <tempUsage><tempUsage limit="10 gb"/></tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>         <transportConnectors>
>             <transportConnector name="openwire" uri="tcp://0.0.0.0:61668"
> />
>             <transportConnector name="vm" uri="vm://localhost" />
>         </transportConnectors>
>     </broker>
>
>     <import resource="jndi.xml"/>
>     <import resource="camel.xml"/>
>
>
> ceposta wrote
> > Did you see any exceptions in the broker logs?
> > Seems like the server-side VMTransport stopped draining its message queue
> > and processing consumer acks... causing the client VMTransport to block
> on
> > a put...and all other consumers using that connection to block...
> >
> >
> > On Mon, Jul 22, 2013 at 3:46 PM, boday &lt;
>
> > ben.oday@
>
> > &gt;wrote:
> >
> >> seeing some thread blocking in our ActiveMQ 5.7 production application
> >> (using
> >> VM transport and AMQ connection pool)...anyone know of any
> >> configuration/known bugs that could be contributing to this?
> >>
> >> seeing 20 thread like this one, WAITING for a single thread to release a
> >> lock on 659ec0a4
> >>
> >> "Camel (camel-1) thread #29 - JmsConsumer[MyQueue]" - Thread t@130
> >>    java.lang.Thread.State: WAITING
> >>         at sun.misc.Unsafe.park(Native Method)
> >>         - waiting to lock <659ec0a4> (a
> >> java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "Camel
> >> (camel-1) thread #13 - JmsConsumer[MyQueue]" t@94
> >>         at
> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> >>         at
> >>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >>         at
> >>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> >>         at
> >>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> >>         at
> >>
> >>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
> >>         at
> >> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
> >>         at
> >>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> >>         at
> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1290)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1284)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1898)
> >>         at
> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2064)
> >>         at
> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2059)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQMessageConsumer.afterMessageIsConsumed(ActiveMQMessageConsumer.java:921)
> >>         - locked <7212432> (a java.util.LinkedList)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:602)
> >>         at
> >>
> >>
> org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>         at java.lang.Thread.run(Thread.java:662)
> >>
> >>    Locked ownable synchronizers:
> >>         - locked <10cf5246> (a
> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >>
> >>
> >> then thiis thread appears to be locking the resource in
> >> question...659ec0a4
> >>
> >>
> >> "Camel (camel-1) thread #13 - JmsConsumer[MyQueue]" - Thread t@94
> >>    java.lang.Thread.State: WAITING
> >>         at sun.misc.Unsafe.park(Native Method)
> >>         - parking to wait for <1ec8dc2f> (a
> >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>         at
> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> >>         at
> >>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> >>         at
> >>
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
> >>         at
> >> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:93)
> >>         at
> >>
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> >>         at
> >>
> >>
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1290)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1284)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1898)
> >>         at
> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2064)
> >>         at
> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2059)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQMessageConsumer.afterMessageIsConsumed(ActiveMQMessageConsumer.java:921)
> >>         - locked <119a6cfe> (a java.util.LinkedList)
> >>         at
> >>
> >>
> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:602)
> >>         at
> >>
> >>
> org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
> >>         at
> >>
> >>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
> >>         at
> >>
> >>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> >>         at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> >>         at java.lang.Thread.run(Thread.java:662)
> >>
> >>    Locked ownable synchronizers:
> >>         - locked <659ec0a4> (a
> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >>
> >>         - locked <3169a0b9> (a
> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/consumer-threads-waiting-on-VMTransport-locks-tp4669580.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
> > --
> > *Christian Posta*
> > http://www.christianposta.com/blog
> > twitter: @christianposta
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/consumer-threads-waiting-on-VMTransport-locks-tp4669580p4669593.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message