activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Lammens (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-4351) Deadlock when unsubscribing durable subscriber
Date Wed, 27 Feb 2013 10:11:13 GMT

     [ https://issues.apache.org/jira/browse/AMQ-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tim Lammens updated AMQ-4351:
-----------------------------

    Description: 
We started using durable subscribers a week ago, and after the 4th durable subscriber unsubscribed
(due to 1 hour inactivity), the system deadlocked. If the "Durable Subscriber Cleanup Timer"
goes of at the wrong time, your entire server locks up. 

Setup: 
Active MQ 5.7.0 with master/slave using JDBC store 
Approx 3 - 5 concurrent durable subscribers 
Approx 5 messages / sec 

Active MQ checks every 1 minute for subscribers which have been offline for 1 hour. 

Locked threads: 
"ActiveMQ Transport: tcp:///79.125.71.104:48082@8090": 
        at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:446) 
        - waiting to lock <0x00000000c6228480> (a org.apache.activemq.broker.region.Topic)

        at org.apache.activemq.broker.region.Topic.send(Topic.java:427) 
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)

        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) 


"ActiveMQ Transport: tcp:///79.125.71.104:47590@8090": 
        at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:142)

        - waiting to lock <0x00000000c66ba050> (a java.lang.Object) 
        at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:243)

        at org.apache.activemq.broker.region.policy.StrictOrderDispatchPolicy.dispatch(StrictOrderDispatchPolicy.java:58)



"ActiveMQ Durable Subscriber Cleanup Timer": 
        at org.apache.activemq.broker.region.Topic.deactivate(Topic.java:288) 
        - waiting to lock <0x00000000c6250670> (a java.util.concurrent.CopyOnWriteArrayList)

        at org.apache.activemq.broker.region.DurableTopicSubscription.deactivate(DurableTopicSubscription.java:184)

        - locked <0x00000000c66ba060> (a java.lang.Object) 
        - locked <0x00000000c66ba050> (a java.lang.Object) 
        at org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:195) 
        at org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:199)

        at org.apache.activemq.broker.region.TopicRegion.doCleanup(TopicRegion.java:99) 


I have attached a patch which fixes the problem.
Since there is only one dispatch policy per Topic, synchronisation can happen on the DispatchPolicy
instead of on the consumers object which causes the deadlock.

  was:

We started using durable subscribers a week ago, and after the 4th durable subscriber unsubscribed
(due to 1 hour inactivity), the system deadlocked. If the "Durable Subscriber Cleanup Timer"
goes of at the wrong time, your entire server locks up. 

Setup: 
Active MQ 5.7.0 with master/slave using JDBC store 
Approx 3 - 5 concurrent durable subscribers 
Approx 5 messages / sec 

Active MQ checks every 1 minute for subscribers which have been offline for 1 hour. 

Locked threads: 
"ActiveMQ Transport: tcp:///79.125.71.104:48082@8090": 
        at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:446) 
        - waiting to lock <0x00000000c6228480> (a org.apache.activemq.broker.region.Topic)

        at org.apache.activemq.broker.region.Topic.send(Topic.java:427) 
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)

        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503) 


"ActiveMQ Transport: tcp:///79.125.71.104:47590@8090": 
        at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:142)

        - waiting to lock <0x00000000c66ba050> (a java.lang.Object) 
        at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:243)

        at org.apache.activemq.broker.region.policy.StrictOrderDispatchPolicy.dispatch(StrictOrderDispatchPolicy.java:58)



"ActiveMQ Durable Subscriber Cleanup Timer": 
        at org.apache.activemq.broker.region.Topic.deactivate(Topic.java:288) 
        - waiting to lock <0x00000000c6250670> (a java.util.concurrent.CopyOnWriteArrayList)

        at org.apache.activemq.broker.region.DurableTopicSubscription.deactivate(DurableTopicSubscription.java:184)

        - locked <0x00000000c66ba060> (a java.lang.Object) 
        - locked <0x00000000c66ba050> (a java.lang.Object) 
        at org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:195) 
        at org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:199)

        at org.apache.activemq.broker.region.TopicRegion.doCleanup(TopicRegion.java:99) 


    
> Deadlock when unsubscribing durable subscriber
> ----------------------------------------------
>
>                 Key: AMQ-4351
>                 URL: https://issues.apache.org/jira/browse/AMQ-4351
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.7.0, 5.8.0
>         Environment: Setup: 
> Active MQ 5.7.0 with master/slave using JDBC store 
> Approx 3 - 5 concurrent durable subscribers 
> Approx 5 messages / sec 
>            Reporter: Tim Lammens
>              Labels: patch
>         Attachments: dispatcher.patch
>
>
> We started using durable subscribers a week ago, and after the 4th durable subscriber
unsubscribed (due to 1 hour inactivity), the system deadlocked. If the "Durable Subscriber
Cleanup Timer" goes of at the wrong time, your entire server locks up. 
> Setup: 
> Active MQ 5.7.0 with master/slave using JDBC store 
> Approx 3 - 5 concurrent durable subscribers 
> Approx 5 messages / sec 
> Active MQ checks every 1 minute for subscribers which have been offline for 1 hour. 
> Locked threads: 
> "ActiveMQ Transport: tcp:///79.125.71.104:48082@8090": 
>         at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:446) 
>         - waiting to lock <0x00000000c6228480> (a org.apache.activemq.broker.region.Topic)

>         at org.apache.activemq.broker.region.Topic.send(Topic.java:427) 
>         at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)

>         at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)

> "ActiveMQ Transport: tcp:///79.125.71.104:47590@8090": 
>         at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:142)

>         - waiting to lock <0x00000000c66ba050> (a java.lang.Object) 
>         at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:243)

>         at org.apache.activemq.broker.region.policy.StrictOrderDispatchPolicy.dispatch(StrictOrderDispatchPolicy.java:58)

> "ActiveMQ Durable Subscriber Cleanup Timer": 
>         at org.apache.activemq.broker.region.Topic.deactivate(Topic.java:288) 
>         - waiting to lock <0x00000000c6250670> (a java.util.concurrent.CopyOnWriteArrayList)

>         at org.apache.activemq.broker.region.DurableTopicSubscription.deactivate(DurableTopicSubscription.java:184)

>         - locked <0x00000000c66ba060> (a java.lang.Object) 
>         - locked <0x00000000c66ba050> (a java.lang.Object) 
>         at org.apache.activemq.broker.region.Topic.deleteSubscription(Topic.java:195)

>         at org.apache.activemq.broker.region.TopicRegion.removeSubscription(TopicRegion.java:199)

>         at org.apache.activemq.broker.region.TopicRegion.doCleanup(TopicRegion.java:99)

> I have attached a patch which fixes the problem.
> Since there is only one dispatch policy per Topic, synchronisation can happen on the
DispatchPolicy instead of on the consumers object which causes the deadlock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message