activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-6601) AMQ with master and slave broker, shutting down slave causes dead-lock
Date Sat, 04 Mar 2017 20:06:45 GMT

    [ https://issues.apache.org/jira/browse/AMQ-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895846#comment-15895846
] 

ASF GitHub Bot commented on AMQ-6601:
-------------------------------------

GitHub user catull opened a pull request:

    https://github.com/apache/activemq/pull/228

    [AMQ-6601] AMQ with master and slave broker, shutting down slave causes a dead-lock

    This PR addresses the issue https://issues.apache.org/jira/browse/AMQ-6601.
    
    The proposed solution is to start the broker asynchronously, but only wait for its starting
up if it is the master (i.e. non-slave).
    
    Closes / fixes issue #6601.
    
    Credit goes to @lichtin for coming up with the solution.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/catull/activemq fix-issue-6601-shutdown-slave-broker

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/activemq/pull/228.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #228
    
----
commit 5de4f788d17f406c713d42a1bd77c15b973ac0de
Author: Carlo Dapor <catull@gmail.com>
Date:   2017-03-04T20:01:34Z

    [AMQ-6601] AMQ with master and slave broker, shutting down slave causes dead-lock.

----


> AMQ with master and slave broker, shutting down slave causes dead-lock
> ----------------------------------------------------------------------
>
>                 Key: AMQ-6601
>                 URL: https://issues.apache.org/jira/browse/AMQ-6601
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.14.0
>            Reporter: Carlo Dapor
>         Attachments: jstack-28657
>
>
> We have 2 karaf instances configured to be activemq brokers, broker-amq (b1) and broker2-amq
(b2).
> They run on the same machine, use KahaDB with file locking.
> It does not matter if b1 or b2 is started first, it becomes the master.
> The other one, the slave, when shutting down - while the master is running - hits a dead-lock;
it must be `kill -9`'d manually in the end.
> We have a classic dead-lock scenario.  I have attached a `jstack` output when the slave
broker is shutting down.
> The race is on between thread #20 and thread #17.
> Thread #17 is in 
> {code:java}
> ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
> {code}
> and thread #20 is in
> {code:java}
> ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
> {code}
> {code}
> "CM Configuration Updater (ManagedServiceFactory Update: factoryPid=[org.apache.activemq.server])"
#20 daemon prio=5 os_prio=0 tid=0x00007f793c160800 nid=0x7084 waiting on condition [0x00007f799819f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at java.lang.Thread.sleep(Thread.java:340)
>         at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
>         at org.apache.activemq.store.SharedFileLocker.doStart(SharedFileLocker.java:83)
>         at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
>         at org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
>         at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
>         at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)
>         at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)
>         at org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)
>         at org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)
>         - locked <0x000000072bd74db0> (a org.apache.activemq.osgi.ActiveMQServiceFactory)
>         at Proxy8890d2d1_e3a3_4b71_a7a0_88810df56856.updated(Unknown Source)
>         at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)
>         at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)
>         at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1597)
>         at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1540)
>         at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:103)
>         at java.lang.Thread.run(Thread.java:745)
>  
> "Thread-4" #19 daemon prio=5 os_prio=0 tid=0x00007f7940002800 nid=0x7081 runnable [0x00007f79984b4000]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>         at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
>         at java.net.ServerSocket.implAccept(ServerSocket.java:545)
>         at java.net.ServerSocket.accept(ServerSocket.java:513)
>         at org.apache.karaf.main.ShutdownSocketThread.run(ShutdownSocketThread.java:56)
>  
> "Thread-3" #18 prio=5 os_prio=0 tid=0x00007f79d0c48800 nid=0x7080 waiting on condition
[0x00007f79985b5000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.karaf.main.Main.doMonitor(Main.java:299)
>         at org.apache.karaf.main.Main.access$100(Main.java:65)
>         at org.apache.karaf.main.Main$1.run(Main.java:275)
>  
> "FelixStartLevel" #17 daemon prio=5 os_prio=0 tid=0x00007f79d0c48000 nid=0x707f waiting
for monitor entry [0x00007f79986b5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.activemq.osgi.ActiveMQServiceFactory.destroy(ActiveMQServiceFactory.java:173)
>         - waiting to lock <0x000000072bd74db0> (a org.apache.activemq.osgi.ActiveMQServiceFactory)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)
>         at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:980)
>         at org.apache.aries.blueprint.container.BeanRecipe.destroy(BeanRecipe.java:887)
>         at org.apache.aries.blueprint.container.BlueprintRepository.destroy(BlueprintRepository.java:329)
>         at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroyComponents(BlueprintContainerImpl.java:765)
>         at org.apache.aries.blueprint.container.BlueprintContainerImpl.tidyupComponents(BlueprintContainerImpl.java:964)
>         at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:909)
>         at org.apache.aries.blueprint.container.BlueprintExtender$3.run(BlueprintExtender.java:325)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at org.apache.aries.blueprint.container.BlueprintExtender.destroyContainer(BlueprintExtender.java:346)
>         at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:238)
>         at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)
>         at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)
>         at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)
>         at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)
>         at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)
>         at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
>         at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
>         at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
>         at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)
>         at org.apache.felix.framework.Felix.stopBundle(Felix.java:2520)
>         at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1309)
>         at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
>         at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message