activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From boday <ben.o...@initekconsulting.com>
Subject Re: consumer threads waiting on VMTransport locks...
Date Wed, 24 Jul 2013 16:40:26 GMT
at this point, I can't reproduce it on demand, just seeing random locking
occur after hours of uptime...


ceposta wrote
> Any chance you can reproduce this?
> 
> 
> On Tue, Jul 23, 2013 at 1:03 AM, boday &lt;

> ben.oday@

> &gt;wrote:
> 
>> no errors in the logs, but we see message flow resume after a server
>> restart...I'm starting to question our connection/server configuration a
>> bit
>> now given this thread dump...
>>
>> our application uses camel-activemq (2.10.1) and runs embedded in an AMQ
>> broker (5.7)....use the same connection factory/pool for all of our apps
>> camel routes that produce/consume from queues, etc...
>>
>> here is my client code connection setup...
>>
>>         ActiveMQConnectionFactory queueConnectionFactory = new
>> ActiveMQConnectionFactory("vm://localhost?create=false");
>>     queueConnectionFactory.setObjectMessageSerializationDefered(true);
>>         queueConnectionFactory.setOptimizedMessageDispatch(true);
>>         queueConnectionFactory.setDispatchAsync(true);
>>         queueConnectionFactory.setUseAsyncSend(true);
>>         queueConnectionFactory.setAlwaysSessionAsync(true);
>>
>>     PooledConnectionFactory pcf = new
>> PooledConnectionFactory(queueConnectionFactory);
>>         pcf.setIdleTimeout(0);
>>
>>         ActiveMQComponent amq =
>> ActiveMQComponent.activeMQComponent(brokerURL);
>>         amq.setConnectionFactory(pcf);
>>         amq.setMaxConcurrentConsumers(20);
>>         amq.setCacheLevelName("CACHE_CONSUMER");
>>         amq.setAcknowledgementModeName("AUTO_ACKNOWLEDGE");
>>         this.getContext().addComponent("activemq", amq);
>>
>> here is my activemq.xml...
>>
>>    
> <broker xmlns="http://activemq.apache.org/schema/core"
>>
>  brokerName="localhost" dataDirectory="../data" useJmx="true">
>>
>>         
> <destinationPolicy>
>>                
> <policyMap>
>>                   
> <policyEntries>
>>                    
> <policyEntry queue=">
> " producerFlowControl="false">
>>                    
> <pendingQueuePolicy>
>>                      
> <vmQueueCursor/>
>>                    
> </pendingQueuePolicy>
>>                   
> </policyEntry>
>>                   
> </policyEntries>
>>                
> </policyMap>
>>         
> </destinationPolicy>
>>
>>         
> <managementContext>
>>             
> <managementContext createConnector="false"/>
>>         
> </managementContext>
>>
>>         
> <networkConnectors>
>>         
> </networkConnectors>
>>
>>         
> <persistenceAdapter>
>>             
> <kahaDB directory="../data/kaha"
>>
>                  enableJournalDiskSyncs="false"
>>                 indexWriteBatchSize="10000"
>>                 indexCacheSize="1000" />
>>         
> </persistenceAdapter>
>>
>>         
> <systemUsage>
>>             
> <systemUsage>
>>                 
> <memoryUsage>
> <memoryUsage limit="4 gb"/>
> </memoryUsage>
>>                 
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
>>                 
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
>>             
> </systemUsage>
>>         
> </systemUsage>
>>
>>         
> <transportConnectors>
>>             
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61668"
>>
>  />
>>             
> <transportConnector name="vm" uri="vm://localhost" />
>>         
> </transportConnectors>
>>     
> </broker>
>>
>>     
> <import resource="jndi.xml"/>
>>     
> <import resource="camel.xml"/>
>>
>>
>> ceposta wrote
>> > Did you see any exceptions in the broker logs?
>> > Seems like the server-side VMTransport stopped draining its message
>> queue
>> > and processing consumer acks... causing the client VMTransport to block
>> on
>> > a put...and all other consumers using that connection to block...
>> >
>> >
>> > On Mon, Jul 22, 2013 at 3:46 PM, boday &lt;
>>
>> > ben.oday@
>>
>> > &gt;wrote:
>> >
>> >> seeing some thread blocking in our ActiveMQ 5.7 production application
>> >> (using
>> >> VM transport and AMQ connection pool)...anyone know of any
>> >> configuration/known bugs that could be contributing to this?
>> >>
>> >> seeing 20 thread like this one, WAITING for a single thread to release
>> a
>> >> lock on 659ec0a4
>> >>
>> >> "Camel (camel-1) thread #29 - JmsConsumer[MyQueue]" - Thread t@130
>> >>    java.lang.Thread.State: WAITING
>> >>         at sun.misc.Unsafe.park(Native Method)
>> >>         - waiting to lock <659ec0a4> (a
>> >> java.util.concurrent.locks.ReentrantLock$NonfairSync) owned by "Camel
>> >> (camel-1) thread #13 - JmsConsumer[MyQueue]" t@94
>> >>         at
>> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>> >>         at
>> >> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>> >>         at
>> >>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1290)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1284)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1898)
>> >>         at
>> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2064)
>> >>         at
>> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2059)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQMessageConsumer.afterMessageIsConsumed(ActiveMQMessageConsumer.java:921)
>> >>         - locked <7212432> (a java.util.LinkedList)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:602)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>         at java.lang.Thread.run(Thread.java:662)
>> >>
>> >>    Locked ownable synchronizers:
>> >>         - locked <10cf5246> (a
>> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> >>
>> >>
>> >> then thiis thread appears to be locking the resource in
>> >> question...659ec0a4
>> >>
>> >>
>> >> "Camel (camel-1) thread #13 - JmsConsumer[MyQueue]" - Thread t@94
>> >>    java.lang.Thread.State: WAITING
>> >>         at sun.misc.Unsafe.park(Native Method)
>> >>         - parking to wait for <1ec8dc2f> (a
>> >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> >>         at
>> >> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>> >>         at
>> >>
>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
>> >>         at
>> >>
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:93)
>> >>         at
>> >>
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1290)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1284)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1898)
>> >>         at
>> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2064)
>> >>         at
>> >> org.apache.activemq.ActiveMQSession.sendAck(ActiveMQSession.java:2059)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQMessageConsumer.afterMessageIsConsumed(ActiveMQMessageConsumer.java:921)
>> >>         - locked <119a6cfe> (a java.util.LinkedList)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:602)
>> >>         at
>> >>
>> >>
>> org.apache.activemq.pool.PooledMessageConsumer.receive(PooledMessageConsumer.java:67)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:429)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
>> >>         at
>> >>
>> >>
>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> >>         at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> >>         at java.lang.Thread.run(Thread.java:662)
>> >>
>> >>    Locked ownable synchronizers:
>> >>         - locked <659ec0a4> (a
>> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> >>
>> >>         - locked <3169a0b9> (a
>> >> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/consumer-threads-waiting-on-VMTransport-locks-tp4669580.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >
>> >
>> >
>> > --
>> > *Christian Posta*
>> > http://www.christianposta.com/blog
>> > twitter: @christianposta
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/consumer-threads-waiting-on-VMTransport-locks-tp4669580p4669593.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> 
> 
> -- 
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta





--
View this message in context: http://activemq.2283324.n4.nabble.com/consumer-threads-waiting-on-VMTransport-locks-tp4669580p4669684.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message