activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sam hendley (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-4361) Deadlock during close while publishing to flow-controlled queue
Date Thu, 07 Mar 2013 15:20:13 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595953#comment-13595953
] 

Sam hendley edited comment on AMQ-4361 at 3/7/13 3:19 PM:
----------------------------------------------------------

So it makes sense to me why the non-producer window version blocks; native socket writes don't
react to thread.interrupt(). I can't think of a way around that problem. The only thing I
would imagine helping is to default to the other strategy and then require people to opt-in
to this more dangerous (and less performant?) mode.

However I was annoyed by the fact I need a Thread.interrupt even with the other version. This
means I need to add an extra level of synchronized bookkeeping to perform a clean shutdown.
I dug into the source code of activemq and the issue is there is a tight wait loop in org.apache.activemq.usage.MemoryUsage
where it only checks the "percentUsed" field. I believe it should also look at the "started"
field and bail if the usage counter has been stopped. I have to imagine that was what the
developers of that class were planning, on the stop method they notify on the same mutex to
wake up all the listeners (presumably so they can bail quickly). I looked at all of the implementations
of Usage and they all seem to have the same issue. Shouldn't they all check "started" and
bail if the condition they are trying to reach can never be reached?
                
      was (Author: shendley):
    So it makes sense to me why the non-producer window version blocks; native socket writes
don't react to thread.interrupt(). I can't think of a way around that problem. The only thing
I would imagine helping is to default to the other strategy and then require people to opt-in
to this more dangerous (and less performant?) mode.

However I was annoyed by the fact I need a Thread.interrupt even with the other version. This
means I need to add an extra level of synchronized bookkeeping to perform a clean shutdown.
I dug into the source code of activemq and the issue is there is a tight wait loop in org.apache.activemq.usage.MemoryUsage
where it only checks the "percentUsed" field. I believe it should also look at the "started"
field and bail if the usage counter has been blocked. I have to imagine that was what the
developers of that class were planning, on the stop method they notify on the same mutex to
wake up all the listeners (presumably so they can bail quickly). I looked at all of the implementations
of Usage and they all seem to have the same issue. Shouldn't they all check "started" and
bail if the condition they are trying to reach can never be reached?
                  
> Deadlock during close while publishing to flow-controlled queue
> ---------------------------------------------------------------
>
>                 Key: AMQ-4361
>                 URL: https://issues.apache.org/jira/browse/AMQ-4361
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: JMS client
>    Affects Versions: 5.6.0, 5.8.0
>         Environment: Windows, Linux, JDK 1.6
>            Reporter: Sam hendley
>
> TestCase on github: 
> https://github.com/samhendley/activemq-close-deadlock-bug
> The deadlock occurs when we are using TcpTransport to a flow-controlled queue and we
then try to gracefully shutdown the application. The close operation hangs forever because
it is trying to send a "close packet" to the server. It never gets the chance to send that
request because the socket is blocked by the publishing thread. This stops my publisher from
shutting down and causes us to orphan threads during shutdown.
> I have verified this bug occurs on atleast activemq 5.6.0 and 5.8.0 and on linux and
windows using JDK 1.6.
> I don't need a fix for the bug necessarily, just a way to gracefully shutdown my publisher
if I get into this state.
> Partial Stack Trace During failure
> "ClosingThread" prio=6 tid=0x045ce000 nid=0xa84 waiting on condition [0x04ddf000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x23fc52d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
>     at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
>     at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>     at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
>     at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>     at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1304)
>     at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1298)
>     at org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1901)
>     at org.apache.activemq.ActiveMQMessageProducer.close(ActiveMQMessageProducer.java:173)
>     at org.activemq.bug.DeadlockDuringCloseTest$2.run(DeadlockDuringCloseTest.java:83)
>     at java.lang.Thread.run(Thread.java:662)
> "PublishingThread" prio=6 tid=0x045cd800 nid=0xb84 runnable [0x04d8f000]
>    java.lang.Thread.State: RUNNABLE
>     at java.net.SocketOutputStream.socketWrite0(Native Method)
>     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>     at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
>     at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>     at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
>     at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:322)
>     at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:304)
>     at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
>     at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
>     at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>     at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>     at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1304)
>     at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1298)
>     at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1782)
>     - locked <0x23faa7d8> (a java.lang.Object)
>     at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:289)
>     at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:224)
>     at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
>     at org.activemq.bug.DeadlockDuringCloseTest$1.run(DeadlockDuringCloseTest.java:63)
>     at java.lang.Thread.run(Thread.java:662)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message