activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Frederik Colardyn (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMQ-4351) Deadlock when unsubscribing durable subscriber
Date Thu, 28 Feb 2013 14:19:15 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589553#comment-13589553
] 

Frederik Colardyn edited comment on AMQ-4351 at 2/28/13 2:18 PM:
-----------------------------------------------------------------

Hi Gary,

It's easy to reproduce, and we've got a unit test for it (see attach), but it's not very well
suited to include in an automated build system as it produces high load over ± 4 minutes
in order to trigger the deadlock. It's reliable though, always deadlocking without the patch.

Scenario:
1. 100 durable consumers, of which 97 go inactive after receiving the first message
2. producer sends messages as fast as possible
3. after 3 minutes, broker tries to clean up the 97 inactive producers => always deadlocks.
No more messages are being sent at all.

We took 100 durable consumers in our test in order to provoke the deadlock quickly, but in
our production system it occurred with only 4 consumers (of which 1 was cleaned up after 1
hour inactivity).
                
      was (Author: frederikcolardyn):
    junit test
                  
> Deadlock when unsubscribing durable subscriber
> ----------------------------------------------
>
>                 Key: AMQ-4351
>                 URL: https://issues.apache.org/jira/browse/AMQ-4351
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.7.0, 5.8.0
>         Environment: Setup: 
> Active MQ 5.7.0 with master/slave using JDBC store 
> Approx 3 - 5 concurrent durable subscribers 
> Approx 5 messages / sec 
>            Reporter: Tim Lammens
>              Labels: patch
>         Attachments: ActiveMQUnsubscribeTest.java, dispatcher.patch
>
>
> We started using durable subscribers a week ago, and after the 4th durable subscriber
unsubscribed (due to 1 hour inactivity), the system deadlocked. If the "Durable Subscriber
Cleanup Timer" goes of at the wrong time, your entire server locks up. 
> Setup: 
> Active MQ 5.7.0 with master/slave using JDBC store 
> Approx 3 - 5 concurrent durable subscribers 
> Approx 5 messages / sec 
> Active MQ checks every 1 minute for subscribers which have been offline for 1 hour. 
> Locked threads: 
> "ActiveMQ Transport: tcp:///79.125.71.104:48082@8090": 
>         at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:446) 
>         - waiting to lock <0x00000000c6228480> (a org.apache.activemq.broker.region.Topic)

>         at org.apache.activemq.broker.region.Topic.send(Topic.java:427) 
>         at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)

>         at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)

> "ActiveMQ Transport: tcp:///79.125.71.104:47590@8090": 
>         at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:142)

>         - waiting to lock <0x00000000c66ba050> (a java.lang.Object) 
>         at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:243)

>         at org.apache.activemq.broker.region.policy.StrictOrderDispatchPolicy.dispatch(StrictOrderDispatchPolicy.java:58)

> "ActiveMQ Durable Subscriber Cleanup Timer": 
>         at org.apache.activemq.broker.region.Topic.deactivate(Topic.java:288) 
>         - waiting to lock <0x00000000c6250670> (a java.util.concurrent.CopyOnWriteArrayList)

>         at org.apache.activemq.broker.region.DurableTopicSubscription.deactivate(DurableTopicSubscription.java:184)

>         - locked <0x00000000c66ba060> (a java.lang.Object) 
>         - locked <0x00000000c66ba050> (a java.lang.Object) 
>         at org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:195)

>         at org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:199)

>         at org.apache.activemq.broker.region.TopicRegion.doCleanup(TopicRegion.java:99)

> I have attached a patch which fixes the problem.
> Since there is only one dispatch policy per Topic, synchronisation can happen on the
DispatchPolicy instead of on the consumers object which causes the deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message