activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Yara (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-4935) Deadlock caused by Re-entrant lock with no locking thread
Date Thu, 12 Dec 2013 23:09:07 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846875#comment-13846875
] 

Michael Yara commented on AMQ-4935:
-----------------------------------

I have discovered that the issue also occurs when only using Node A in a pure vm transport
environment with ObjectMessageSerializationDefered=true and copyMessageOnSend=false on all
connections. 

Looking through the ActiveMQ source it looks as if the object contents are still serialized
despite ObjectMessageSerializationDefered=true on a call to ActiveMQObjectMessage.beforeMarshall(WireFormat
wireFormat) which ends up calling ActiveMQObjectMessage.storeContent(). This call stack seems
to be triggered by calls to the tcp transport methods. I was expecting that my configuration
would prevent serialization entirely if not needed. I do not understand why TCP transport
methods are still called despite the use of vm transport. I am assuming there is a reason
for the calls to the tcp transport methods and serialization to still occur in this situation.

> Deadlock caused by Re-entrant lock with no locking thread
> ---------------------------------------------------------
>
>                 Key: AMQ-4935
>                 URL: https://issues.apache.org/jira/browse/AMQ-4935
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.0
>         Environment: Oracle JRE 1.7.0 update 45
>            Reporter: Michael Yara
>
> We have a distributed application that uses ActiveMQ for communication between nodes.
We are currently using ActiveMQ 5.9.0 running with Oracle JRE 1.7.0 update 45. 
> We are seeing some strange deadlock behavior in the ActiveMQ ThreadPoolExecutor when
trying to serialize instances of ConcurrentHashMap. The thread dump can be seen at the following
link:
> http://pastebin.com/wmNxSF4p
> We can see that there are two threads that are blocked waiting for a lock on some of
the segments in a ConcurrentHashMap yet the thread dump does not indicate any threads that
currently hold a lock on those objects. This lock does not occur all the time and can take
around 5-10 messages before the deadlock occurs. I have been able to recreate this issue on
two different physical machines.
> Our test environment that generated this thread dump is as follows. Node A is the instance
that thread dump was taken from. For testing both nodes are on the same physical machine.
> Node A:
> * In its own JVM instance.
> * Runs an embedded ActiveMQ broker.
> * Performs communication via vm transport and sends ObjectMessages via a TopicConnection.
> * Has copyMessageOnSend=false
> * Has ObjectMessageSerializationDefered=true
> Node B:
> * In its own JVM instance.
> * Connects to Node A's broker via tcp transport.
> * Sends BytesMessages serialized with Kryo and sent via a TopicConnection.
> * Has copyMessageOnSend=false
> * Has ObjectMessageSerializationDefered=false
> Both nodes are set up to detect the type of incoming messages and deserialize them appropriately
with kryo if it receives a BytesMessage or default serialization if it receives an ObjectMessage
(handled by ActiveMQ).
> The only thing I have found resembling this issue is the following link. I am no sure
if it is related to this exact problem or not.
> http://appcrawler.com/wordpress/2013/05/06/one-way-to-tell-if-a-thread-pool-is-hung/



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message