brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BROOKLYN-66) Deadlock in group.addMember() when unmanaging other member of group
Date Sat, 06 Sep 2014 22:42:28 GMT

    [ https://issues.apache.org/jira/browse/BROOKLYN-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124690#comment-14124690
] 

ASF GitHub Bot commented on BROOKLYN-66:
----------------------------------------

Github user grkvlt commented on the pull request:

    https://github.com/apache/incubator-brooklyn/pull/151#issuecomment-54730933
  
    @aledsage I should really have added a test, so thanks for that. The rest looks good also,
recommend merging.


> Deadlock in group.addMember() when unmanaging other member of group
> -------------------------------------------------------------------
>
>                 Key: BROOKLYN-66
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-66
>             Project: Brooklyn
>          Issue Type: Bug
>    Affects Versions: 0.7.0-M1
>            Reporter: Aled Sage
>
> When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` lots of times,
I hit this deadlock.
> In brief...
> When method is synchronized, hit deadlock: 
> 1. thread called unmanage() on a member of a group, so we got the lock on LocalEntityManager

>    and called group.removeMember; this ties to synchronize on AbstractGroupImpl.members

> 2. another thread was doing AbstractGroupImpl.addMember, which is synchronized on AbstractGroupImpl.members;
>    it tries to call Entities.manage(child) which calls LocalEntityManager.getEntity(),
which is
>    synchronized on this.
> We MUST NOT call alien code from within the management framework while holding locks.

> The AbstractGroup.removeMember is effectively alien because a user could override it,
and because
> it is entity specific. Therefore this should be fixed in LocalEntityManager.
> Found one Java-level deadlock:
> =============================
> "brooklyn-execmanager-Lol69DXC-10":
>   waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a brooklyn.management.internal.LocalEntityManager),
>   which is held by "brooklyn-execmanager-Lol69DXC-2"
> "brooklyn-execmanager-Lol69DXC-2":
>   waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a brooklyn.util.collections.SetFromLiveMap),
>   which is held by "brooklyn-execmanager-Lol69DXC-10"
> Java stack information for the threads listed above:
> ===================================================
> "brooklyn-execmanager-Lol69DXC-10":
>         at brooklyn.management.internal.LocalEntityManager.getEntity(LocalEntityManager.java:198)
>         - waiting to lock <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
>         at brooklyn.management.internal.LocalEntityManager.isManaged(LocalEntityManager.java:207)
>         at brooklyn.management.internal.LocalEntityManager.manage(LocalEntityManager.java:235)
>         at brooklyn.entity.basic.Entities.manage(Entities.java:768)
>         at brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGroupImpl.java:132)
>         - locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
>         at com.sun.proxy.$Proxy57.addMember(Unknown Source)
>         at brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes(DynamicClusterImpl.java:683)
>         at brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(DynamicClusterImpl.java:670)
>         at brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(DynamicClusterImpl.java:615)
>         at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:484)
>         - locked <7f34800e8> (a [Ljava.lang.Object;)
>         at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:474)
>         - locked <7f34800e8> (a [Ljava.lang.Object;)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
>         at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
>         at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
>         at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
>         at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
>         at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
>         at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
>         at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
>         at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:695)
> "brooklyn-execmanager-Lol69DXC-2":
>         at brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractGroupImpl.java:148)
>         - waiting to lock <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
>         at com.sun.proxy.$Proxy57.removeMember(Unknown Source)
>         at brooklyn.management.internal.LocalEntityManager.unmanageNonRecursive(LocalEntityManager.java:408)
>         - locked <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
>         at brooklyn.management.internal.LocalEntityManager.unmanage(LocalEntityManager.java:284)
>         at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851)
>         at brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicClusterImpl.java:836)
>         at brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicClusterImpl.java:609)
>         at brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterImpl.java:393)
>         at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
>         at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
>         at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
>         at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:102)
>         at com.sun.proxy.$Proxy56.stop(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:166)
>         at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
>         at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
>         at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
>         at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:695)
> Found 1 deadlock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message