brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BROOKLYN-66) Deadlock in group.addMember() when unmanaging other member of group
Date Fri, 05 Sep 2014 20:39:28 GMT
Aled Sage created BROOKLYN-66:
---------------------------------

             Summary: Deadlock in group.addMember() when unmanaging other member of group
                 Key: BROOKLYN-66
                 URL: https://issues.apache.org/jira/browse/BROOKLYN-66
             Project: Brooklyn
          Issue Type: Bug
    Affects Versions: 0.7.0-M1
            Reporter: Aled Sage


When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` lots of times, I
hit this deadlock.

In brief...

When method is synchronized, hit deadlock: 
1. thread called unmanage() on a member of a group, so we got the lock on LocalEntityManager

   and called group.removeMember; this ties to synchronize on AbstractGroupImpl.members 
2. another thread was doing AbstractGroupImpl.addMember, which is synchronized on AbstractGroupImpl.members;
   it tries to call Entities.manage(child) which calls LocalEntityManager.getEntity(), which
is
   synchronized on this.

We MUST NOT call alien code from within the management framework while holding locks. 
The AbstractGroup.removeMember is effectively alien because a user could override it, and
because
it is entity specific. Therefore this should be fixed in LocalEntityManager.

Found one Java-level deadlock:
=============================
"brooklyn-execmanager-Lol69DXC-10":
  waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a brooklyn.management.internal.LocalEntityManager),
  which is held by "brooklyn-execmanager-Lol69DXC-2"
"brooklyn-execmanager-Lol69DXC-2":
  waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a brooklyn.util.collections.SetFromLiveMap),
  which is held by "brooklyn-execmanager-Lol69DXC-10"

Java stack information for the threads listed above:
===================================================
"brooklyn-execmanager-Lol69DXC-10":
        at brooklyn.management.internal.LocalEntityManager.getEntity(LocalEntityManager.java:198)
        - waiting to lock <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
        at brooklyn.management.internal.LocalEntityManager.isManaged(LocalEntityManager.java:207)
        at brooklyn.management.internal.LocalEntityManager.manage(LocalEntityManager.java:235)
        at brooklyn.entity.basic.Entities.manage(Entities.java:768)
        at brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGroupImpl.java:132)
        - locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
        at com.sun.proxy.$Proxy57.addMember(Unknown Source)
        at brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes(DynamicClusterImpl.java:683)
        at brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(DynamicClusterImpl.java:670)
        at brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(DynamicClusterImpl.java:615)
        at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:484)
        - locked <7f34800e8> (a [Ljava.lang.Object;)
        at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:474)
        - locked <7f34800e8> (a [Ljava.lang.Object;)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
        at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
        at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
        at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
        at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
        at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
        at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
        at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
        at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
        at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
        at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:695)
"brooklyn-execmanager-Lol69DXC-2":
        at brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractGroupImpl.java:148)
        - waiting to lock <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
        at com.sun.proxy.$Proxy57.removeMember(Unknown Source)
        at brooklyn.management.internal.LocalEntityManager.unmanageNonRecursive(LocalEntityManager.java:408)
        - locked <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
        at brooklyn.management.internal.LocalEntityManager.unmanage(LocalEntityManager.java:284)
        at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851)
        at brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicClusterImpl.java:836)
        at brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicClusterImpl.java:609)
        at brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterImpl.java:393)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
        at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
        at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
        at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
        at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
        at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
        at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:102)
        at com.sun.proxy.$Proxy56.stop(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:166)
        at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
        at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
        at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
        at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:695)

Found 1 deadlock.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message