Return-Path: X-Original-To: apmail-brooklyn-dev-archive@minotaur.apache.org Delivered-To: apmail-brooklyn-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AAC0411FE9 for ; Fri, 5 Sep 2014 20:40:55 +0000 (UTC) Received: (qmail 32895 invoked by uid 500); 5 Sep 2014 20:40:55 -0000 Delivered-To: apmail-brooklyn-dev-archive@brooklyn.apache.org Received: (qmail 32866 invoked by uid 500); 5 Sep 2014 20:40:55 -0000 Mailing-List: contact dev-help@brooklyn.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@brooklyn.incubator.apache.org Delivered-To: mailing list dev@brooklyn.incubator.apache.org Received: (qmail 32855 invoked by uid 99); 5 Sep 2014 20:40:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2014 20:40:55 +0000 X-ASF-Spam-Status: No, hits=-2001.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 05 Sep 2014 20:40:31 +0000 Received: (qmail 32360 invoked by uid 99); 5 Sep 2014 20:40:29 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2014 20:40:29 +0000 Date: Fri, 5 Sep 2014 20:40:29 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@brooklyn.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BROOKLYN-66) Deadlock in group.addMember() when unmanaging other member of group MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/BROOKLYN-66?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D141= 23536#comment-14123536 ]=20 ASF GitHub Bot commented on BROOKLYN-66: ---------------------------------------- GitHub user aledsage opened a pull request: https://github.com/apache/incubator-brooklyn/pull/151 BROOKLYN-66: fix deadlock (and various others) =20 You can merge this pull request into a Git repository by running: $ git pull https://github.com/aledsage/incubator-brooklyn fix/deadlock Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-brooklyn/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 =20 ---- commit 7ea0a6578bde8bad35da175593cbd714bb749ce5 Author: Aled Sage Date: 2014-09-05T18:08:01Z BROOKLYN-66: Fix deadlock in LocalEntityManager commit 112193a9b84c932e992e6a6cef7414abf48e10bf Author: Aled Sage Date: 2014-09-05T18:22:32Z Fix concurrency issue stopping cluster =20 - when resizing to zero, we shrink by currentSize. If someone concurrently stops an entity, then don=E2=80=99t worry that we were o= nly able to stop (currentSize-1) entities. Just continue. commit ccb92a88b0b82f1b2a486debd5965dc301168159 Author: Aled Sage Date: 2014-09-05T18:55:18Z Fix failing ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember commit a49280957f5515de54ac6961eec522f16e04130a Author: Aled Sage Date: 2014-09-05T19:02:12Z Change EntityAdjuncts.findWithUniqueTag to tryFindWithUniqueTag =20 - Make it consistent with guava naming commit ac567c16ed39a14deca22075f0ec29314c2fe8b6 Author: Aled Sage Date: 2014-09-05T19:10:32Z Move some tests form AbstractEntityLegacyTest to EntitySpecTest commit 0447fbbd13d4f6fbdc2e3f5aefa6f8ec8c95c65d Author: Aled Sage Date: 2014-09-05T19:14:02Z Simplify HighAvailabilityManagerSplitBrainTest commit 2503de7bb6c993b5dc964d78004262ff2d784916 Author: Aled Sage Date: 2014-09-05T19:17:36Z CatalogLibrariesDo: minor tidy ---- > Deadlock in group.addMember() when unmanaging other member of group > ------------------------------------------------------------------- > > Key: BROOKLYN-66 > URL: https://issues.apache.org/jira/browse/BROOKLYN-66 > Project: Brooklyn > Issue Type: Bug > Affects Versions: 0.7.0-M1 > Reporter: Aled Sage > > When running `ServiceReplacerTest.testSetsOnFireWhenFailToReplaceMember` = lots of times, I hit this deadlock. > In brief... > When method is synchronized, hit deadlock:=20 > 1. thread called unmanage() on a member of a group, so we got the lock on= LocalEntityManager=20 > and called group.removeMember; this ties to synchronize on AbstractGro= upImpl.members=20 > 2. another thread was doing AbstractGroupImpl.addMember, which is synchro= nized on AbstractGroupImpl.members; > it tries to call Entities.manage(child) which calls LocalEntityManager= .getEntity(), which is > synchronized on this. > We MUST NOT call alien code from within the management framework while ho= lding locks.=20 > The AbstractGroup.removeMember is effectively alien because a user could = override it, and because > it is entity specific. Therefore this should be fixed in LocalEntityManag= er. > Found one Java-level deadlock: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > "brooklyn-execmanager-Lol69DXC-10": > waiting to lock monitor 7f9774943fc0 (object 7f3303f70, a brooklyn.mana= gement.internal.LocalEntityManager), > which is held by "brooklyn-execmanager-Lol69DXC-2" > "brooklyn-execmanager-Lol69DXC-2": > waiting to lock monitor 7f977a88b5f0 (object 7f36e75d0, a brooklyn.util= .collections.SetFromLiveMap), > which is held by "brooklyn-execmanager-Lol69DXC-10" > Java stack information for the threads listed above: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > "brooklyn-execmanager-Lol69DXC-10": > at brooklyn.management.internal.LocalEntityManager.getEntity(Loca= lEntityManager.java:198) > - waiting to lock <7f3303f70> (a brooklyn.management.internal.Loc= alEntityManager) > at brooklyn.management.internal.LocalEntityManager.isManaged(Loca= lEntityManager.java:207) > at brooklyn.management.internal.LocalEntityManager.manage(LocalEn= tityManager.java:235) > at brooklyn.entity.basic.Entities.manage(Entities.java:768) > at brooklyn.entity.basic.AbstractGroupImpl.addMember(AbstractGrou= pImpl.java:132) > - locked <7f36e75d0> (a brooklyn.util.collections.SetFromLiveMap) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImp= l.java:104) > at com.sun.proxy.$Proxy57.addMember(Unknown Source) > at brooklyn.entity.group.DynamicClusterImpl.quarantineFailedNodes= (DynamicClusterImpl.java:683) > at brooklyn.entity.group.DynamicClusterImpl.addInEachLocation(Dyn= amicClusterImpl.java:670) > at brooklyn.entity.group.DynamicClusterImpl.addInSingleLocation(D= ynamicClusterImpl.java:615) > at brooklyn.entity.group.DynamicClusterImpl.replaceMember(Dynamic= ClusterImpl.java:484) > - locked <7f34800e8> (a [Ljava.lang.Object;) > at brooklyn.entity.group.DynamicClusterImpl.replaceMember(Dynamic= ClusterImpl.java:474) > - locked <7f34800e8> (a [Ljava.lang.Object;) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMetho= d.java:90) > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047= ) > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877) > at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaCla= ss.java:149) > at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Sourc= e) > at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall= (CallSiteArray.java:42) > at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Sourc= e) > at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(Groovy= JavaMethods.groovy:144) > at brooklyn.management.internal.AbstractManagementContext.invokeE= ffectorMethodLocal(AbstractManagementContext.java:251) > at brooklyn.management.internal.AbstractManagementContext.invokeE= ffectorMethodSync(AbstractManagementContext.java:274) > at brooklyn.management.internal.EffectorUtils.invokeMethodEffecto= r(EffectorUtils.java:238) > at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:= 149) > at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.j= ava:64) > at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffect= or.java:83) > at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSe= quentialTask.java:318) > at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecution= Manager.java:389) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:895) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > "brooklyn-execmanager-Lol69DXC-2": > at brooklyn.entity.basic.AbstractGroupImpl.removeMember(AbstractG= roupImpl.java:148) > - waiting to lock <7f36e75d0> (a brooklyn.util.collections.SetFro= mLiveMap) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImp= l.java:104) > at com.sun.proxy.$Proxy57.removeMember(Unknown Source) > at brooklyn.management.internal.LocalEntityManager.unmanageNonRec= ursive(LocalEntityManager.java:408) > - locked <7f3303f70> (a brooklyn.management.internal.LocalEntityM= anager) > at brooklyn.management.internal.LocalEntityManager.unmanage(Local= EntityManager.java:284) > at brooklyn.entity.basic.Entities.unmcaanage(Entities.java:851) > at brooklyn.entity.group.DynamicClusterImpl.discardNode(DynamicCl= usterImpl.java:836) > at brooklyn.entity.group.DynamicClusterImpl.shrink(DynamicCluster= Impl.java:609) > at brooklyn.entity.group.DynamicClusterImpl.stop(DynamicClusterIm= pl.java:393) > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMetho= d.java:90) > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047= ) > at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877) > at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaCla= ss.java:149) > at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Sourc= e) > at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall= (CallSiteArray.java:42) > at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Sourc= e) > at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(Groovy= JavaMethods.groovy:144) > at brooklyn.management.internal.AbstractManagementContext.invokeE= ffectorMethodLocal(AbstractManagementContext.java:251) > at brooklyn.management.internal.AbstractManagementContext.invokeE= ffectorMethodSync(AbstractManagementContext.java:274) > at brooklyn.management.internal.EffectorUtils.invokeMethodEffecto= r(EffectorUtils.java:238) > at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImp= l.java:102) > at com.sun.proxy.$Proxy56.stop(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:= 166) > at brooklyn.entity.basic.AbstractEffector.call(AbstractEffector.j= ava:64) > at brooklyn.entity.basic.AbstractEffector$1$1.call(AbstractEffect= or.java:83) > at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSe= quentialTask.java:318) > at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecution= Manager.java:389) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP= oolExecutor.java:895) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:918) > at java.lang.Thread.run(Thread.java:695) > Found 1 deadlock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)