brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BROOKLYN-66) Deadlock in group.addMember() when unmanaging other member of group
Date Fri, 05 Sep 2014 20:40:29 GMT

    [ https://issues.apache.org/jira/browse/BROOKLYN-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123536#comment-14123536
] 

ASF GitHub Bot commented on BROOKLYN-66:
----------------------------------------

GitHub user aledsage opened a pull request:

    https://github.com/apache/incubator-brooklyn/pull/151

    BROOKLYN-66: fix deadlock (and various others)

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aledsage/incubator-brooklyn fix/deadlock

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-brooklyn/pull/151.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #151
    
----
commit 7ea0a6578bde8bad35da175593cbd714bb749ce5
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T18:08:01Z

    BROOKLYN-66: Fix deadlock in LocalEntityManager

commit 112193a9b84c932e992e6a6cef7414abf48e10bf
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T18:22:32Z

    Fix concurrency issue stopping cluster
    
    - when resizing to zero, we shrink by currentSize. If someone
      concurrently stops an entity, then don’t worry that we were only
      able to stop (currentSize-1) entities. Just continue.

commit ccb92a88b0b82f1b2a486debd5965dc301168159
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T18:55:18Z

    Fix failing ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember

commit a49280957f5515de54ac6961eec522f16e04130a
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T19:02:12Z

    Change EntityAdjuncts.findWithUniqueTag to tryFindWithUniqueTag
    
    - Make it consistent with guava naming

commit ac567c16ed39a14deca22075f0ec29314c2fe8b6
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T19:10:32Z

    Move some tests form AbstractEntityLegacyTest to EntitySpecTest

commit 0447fbbd13d4f6fbdc2e3f5aefa6f8ec8c95c65d
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T19:14:02Z

    Simplify HighAvailabilityManagerSplitBrainTest

commit 2503de7bb6c993b5dc964d78004262ff2d784916
Author: Aled Sage <aled.sage@gmail.com>
Date:   2014-09-05T19:17:36Z

    CatalogLibrariesDo: minor tidy

----


> Deadlock in group.addMember() when unmanaging other member of group
> -------------------------------------------------------------------
>
>                 Key: BROOKLYN-66
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-66
>             Project: Brooklyn
>          Issue Type: Bug
>    Affects Versions: 0.7.0-M1
>            Reporter: Aled Sage
>
> When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` lots of times,
I hit this deadlock.
> In brief...
> When method is synchronized, hit deadlock: 
> 1. thread called unmanage() on a member of a group, so we got the lock on LocalEntityManager

>    and called group.removeMember; this ties to synchronize on AbstractGroupImpl.members

> 2. another thread was doing AbstractGroupImpl.addMember, which is synchronized on AbstractGroupImpl.members;
>    it tries to call Entities.manage(child) which calls LocalEntityManager.getEntity(),
which is
>    synchronized on this.
> We MUST NOT call alien code from within the management framework while holding locks.

> The AbstractGroup.removeMember is effectively alien because a user could override it,
and because
> it is entity specific. Therefore this should be fixed in LocalEntityManager.
> Found one Java-level deadlock:
> =============================
> "brooklyn-execmanager-Lol69DXC-10":
>   waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a brooklyn.management.internal.LocalEntityManager),
>   which is held by "brooklyn-execmanager-Lol69DXC-2"
> "brooklyn-execmanager-Lol69DXC-2":
>   waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a brooklyn.util.collections.SetFromLiveMap),
>   which is held by "brooklyn-execmanager-Lol69DXC-10"
> Java stack information for the threads listed above:
> ===================================================
> "brooklyn-execmanager-Lol69DXC-10":
>         at brooklyn.management.internal.LocalEntityManager.getEntity(LocalEntityManager.java:198)
>         - waiting to lock <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
>         at brooklyn.management.internal.LocalEntityManager.isManaged(LocalEntityManager.java:207)
>         at brooklyn.management.internal.LocalEntityManager.manage(LocalEntityManager.java:235)
>         at brooklyn.entity.basic.Entities.manage(Entities.java:768)
>         at brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGroupImpl.java:132)
>         - locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
>         at com.sun.proxy.$Proxy57.addMember(Unknown Source)
>         at brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes(DynamicClusterImpl.java:683)
>         at brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(DynamicClusterImpl.java:670)
>         at brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(DynamicClusterImpl.java:615)
>         at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:484)
>         - locked <7f34800e8> (a [Ljava.lang.Object;)
>         at brooklyn.entity.group.DynamicClusterImpl.replaceMember(DynamicClusterImpl.java:474)
>         - locked <7f34800e8> (a [Ljava.lang.Object;)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
>         at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
>         at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
>         at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
>         at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
>         at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
>         at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
>         at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
>         at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:695)
> "brooklyn-execmanager-Lol69DXC-2":
>         at brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractGroupImpl.java:148)
>         - waiting to lock <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:104)
>         at com.sun.proxy.$Proxy57.removeMember(Unknown Source)
>         at brooklyn.management.internal.LocalEntityManager.unmanageNonRecursive(LocalEntityManager.java:408)
>         - locked <7f3303f70> (a brooklyn.management.internal.LocalEntityManager)
>         at brooklyn.management.internal.LocalEntityManager.unmanage(LocalEntityManager.java:284)
>         at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851)
>         at brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicClusterImpl.java:836)
>         at brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicClusterImpl.java:609)
>         at brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterImpl.java:393)
>         at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
>         at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
>         at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
>         at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
>         at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
>         at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:251)
>         at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:274)
>         at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:238)
>         at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:102)
>         at com.sun.proxy.$Proxy56.stop(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:166)
>         at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.java:64)
>         at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffector.java:83)
>         at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
>         at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:695)
> Found 1 deadlock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message