Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7401A177A5 for ; Sun, 2 Nov 2014 15:21:35 +0000 (UTC) Received: (qmail 72233 invoked by uid 500); 2 Nov 2014 15:21:35 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 72207 invoked by uid 500); 2 Nov 2014 15:21:35 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 72198 invoked by uid 500); 2 Nov 2014 15:21:35 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 72195 invoked by uid 99); 2 Nov 2014 15:21:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Nov 2014 15:21:35 +0000 Date: Sun, 2 Nov 2014 15:21:35 +0000 (UTC) From: "Animesh Chaturvedi (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-7799) [Accounts] Account deletion failed to remove all the entities related to it in case of failure in deleting one entity MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-7799?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Animesh Chaturvedi updated CLOUDSTACK-7799: ------------------------------------------- Assignee: Sanjay Tripathi > [Accounts] Account deletion failed to remove all the entities related to = it in case of failure in deleting one entity > -------------------------------------------------------------------------= -------------------------------------------- > > Key: CLOUDSTACK-7799 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-779= 9 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Management Server > Affects Versions: 4.5.0 > Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]# = cloudstack-sccs > 385c4f673dfbd1fd326e539625e2c06db4cdc27d > Reporter: Sanjeev N > Assignee: Sanjay Tripathi > Priority: Critical > Fix For: 4.5.0 > > Attachments: cloud.dmp, management-server.rar > > > [Critical][Accounts] Account deletion failed to remove all the entities r= elated to it in case of failure in deleting one entity > Steps to Reproduce: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 1.Bring up CS in advanced zone with one vmware cluster > 2.Create a user account=20 > 3.With the user account deploy few (4-5) vms=20 > 4.Take snapshots on all of the vms root disks > 5.Deploy another vm and simulate snapshot failure operation(In my case I = tried snapshot operation with quiesce option set to true) so that snapshot = will be in "Allocated" state > 6.Delete this user account > Expected Behavior: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Whatever may the state of the snapshot account deletion shall clean all t= he objects related that account > Actual Behavior: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Account deletion started deleting snapshots. Since one of the snapshots i= s in Allocated state it failed to delete that snapshot and the Account clea= n up job ended there. So vms and networks related to the accounts are still= there but the account was deleted. > Here is the log snippet: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c1= 6bc72) =3D=3D=3DSTART=3D=3D=3D 10.252.193.8 -- GET command=3DdeleteAccoun= t&response=3Djson&sessionkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=3D83f26103= -9d83-4e4e-8bcc-c282c2acb498&_=3D1414470667367 > 2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina= -exec-5:ctx-1c16bc72 ctx-51140d28) submit async job-53, details: AsyncJobVO= {id:53, userId: 2, accountId: 2, instanceType: Account, instanceId: null, = cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdI= nfo: {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessio= nkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.A= ccount\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT= .DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f= 26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"11= 8"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, r= esult: null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null= , lastPolled: null, created: null} > 2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c1= 6bc72 ctx-51140d28) =3D=3D=3DEND=3D=3D=3D 10.252.193.8 -- GET command=3Dd= eleteAccount&response=3Djson&sessionkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id= =3D83f26103-9d83-4e4e-8bcc-c282c2acb498&_=3D1414470667367 > 2014-10-28 15:16:38,996 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53) Add job-53 into job monitoring > 2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53) Executing AsyncJobVO {id:53, userId: 2, ac= countId: 2, instanceType: Account, instanceId: null, cmd: org.apache.clouds= tack.api.command.admin.account.DeleteAccountCmd, cmdInfo: {"response":"json= ","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKi= spQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d= 83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"= 2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c= 282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"}, cmdVersion: 0, st= atus: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:= 6637838401571, completeMsid: null, lastUpdated: null, lastPolled: null, cr= eated: null} > 2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executo= r-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Removed account 4 > 2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292948: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/5","wait":0}}] } > 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:= null) Seq 4-4574812796478292948: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tr= ue,"wait":0}}] } > 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292948: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 5 under account: 4 > 2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-12:null) Seq 4-4574812796478292948: No more commands found > 2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for= snapshot: 1 > 2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 1 doesn't have = children, so it's ok to delete it and its parents > 2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292949: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache= .cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.= storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89= -d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4= f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apac= he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e= 55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7"= ,"path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesy= stem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=3DPrimary&STOREUUI= D=3Dadc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":21474836= 48,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInf= o":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e5= 5fe7db3847af9] i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] = 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}"= ,"format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorTy= pe":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10= .147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA"= ,"name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiesc= evm":false,"physicalSize":0}},"wait":0}}] } > 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (AgentManager-Handler-17:= null) Seq 4-4574812796478292949: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":tru= e,"details":"snapshot directory a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn'= t exist","wait":0}}] } > 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292949: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } } > 2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292950: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/7","wait":0}}] } > 2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452= a9eca) =3D=3D=3DSTART=3D=3D=3D 10.252.193.8 -- GET command=3DqueryAsyncJo= bResult&jobId=3D2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=3Djson&sessio= nkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=3D1414470670595 > 2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452= a9eca ctx-9e36e958) =3D=3D=3DEND=3D=3D=3D 10.252.193.8 -- GET command=3Dq= ueryAsyncJobResult&jobId=3D2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=3D= json&sessionkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=3D1414470670595 > 2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:= null) Seq 4-4574812796478292950: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tr= ue,"wait":0}}] } > 2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292950: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 7 under account: 4 > 2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-15:null) Seq 4-4574812796478292950: No more commands found > 2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for= snapshot: 2 > 2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 2 doesn't have = children, so it's ok to delete it and its parents > 2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292951: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache= .cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.= storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a= -869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639= b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apac= he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e= 55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7"= ,"path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesy= stem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=3DPrimary&STOREUUI= D=3Dadc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":21474836= 48,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInf= o":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e5= 5fe7db3847af9] i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] = 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}"= ,"format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorTy= pe":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10= .147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA"= ,"name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiesc= evm":false,"physicalSize":0}},"wait":0}}] } > 2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request] (AgentManager-Handler-16:= null) Seq 4-4574812796478292951: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":tru= e,"details":"snapshot directory ecb858aa-1656-4594-996a-869f1695bce2 doesn'= t exist","wait":0}}] } > 2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292951: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } } > 2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292952: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/8","wait":0}}] } > 2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request] (AgentManager-Handler-19:= null) Seq 4-4574812796478292952: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tr= ue,"wait":0}}] } > 2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292952: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 8 under account: 4 > 2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-19:null) Seq 4-4574812796478292952: No more commands found > 2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for= snapshot: 3 > 2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 3 doesn't have = children, so it's ok to delete it and its parents > 2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292953: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache= .cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.= storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b= -cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebf= f64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apac= he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e= 55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7"= ,"path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesy= stem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=3DPrimary&STOREUUI= D=3Dadc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":21474836= 48,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInf= o":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e5= 5fe7db3847af9] i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] = 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}"= ,"format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorTy= pe":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10= .147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA"= ,"name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiesc= evm":false,"physicalSize":0}},"wait":0}}] } > 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:= null) Seq 4-4574812796478292953: Processing: { Ans: , MgmtId: 663783840157= 1, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":tru= e,"details":"snapshot directory 657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn'= t exist","wait":0}}] } > 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292953: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } } > 2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292954: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/9","wait":0}}] } > 2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ct= x-6ecc0a29) AutoScaling Monitor is running... > 2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:n= ull) Seq 4-4574812796478292954: Processing: { Ans: , MgmtId: 6637838401571= , via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tru= e,"wait":0}}] } > 2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292954: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 9 under account: 4 > 2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-2:null) Seq 4-4574812796478292954: No more commands found > 2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for= snapshot: 4 > 2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 4 doesn't have = children, so it's ok to delete it and its parents > 2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292955: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache= .cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.= storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572= -09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9= c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apac= he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e= 55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7"= ,"path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesy= stem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=3DPrimary&STOREUUI= D=3Dadc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":21474836= 48,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInf= o":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e5= 5fe7db3847af9] i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] = 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}"= ,"format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorTy= pe":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10= .147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA"= ,"name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiesc= evm":false,"physicalSize":0}},"wait":0}}] } > 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:n= ull) Seq 4-4574812796478292955: Processing: { Ans: , MgmtId: 6637838401571= , via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true= ,"details":"snapshot directory e7d15b83-06be-4596-9572-09478415e776 doesn't= exist","wait":0}}] } > 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292955: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } } > 2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292956: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/10","wait":0}}] } > 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:n= ull) Seq 4-4574812796478292956: Processing: { Ans: , MgmtId: 6637838401571= , via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tru= e,"wait":0}}] } > 2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-3:null) Seq 4-4574812796478292956: No more commands found > 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292956: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 10 under account: 4 > 2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for= snapshot: 5 > 2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-= Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 5 doesn't have = children, so it's ok to delete it and its parents > 2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292957: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache= .cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.= storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0= e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40= dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org= .apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3d= b8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.= 28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkF= ilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=3DPrimary&STO= REUUID=3Dadc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":53= 68709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId= ":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMwa= re"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7= /export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_201= 41027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSi= ze":0}},"wait":0}}] } > 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:n= ull) Seq 4-4574812796478292957: Processing: { Ans: , MgmtId: 6637838401571= , via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true= ,"details":"snapshot directory 1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't= exist","wait":0}}] } > 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292957: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } } > 2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292958: Sending { Cmd , M= gmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.= agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsT= O":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}}= ,"directory":"snapshots/4/11","wait":0}}] } > 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:n= ull) Seq 4-4574812796478292958: Processing: { Ans: , MgmtId: 6637838401571= , via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tru= e,"details":"snapshot directory 11 doesn't exist","wait":0}}] } > 2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handle= r-4:null) Seq 4-4574812796478292958: No more commands found > 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-= 0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292958: Received: { Ans: = , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } } > 2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume:= 11 under account: 4 > 2014-10-28 15:16:44,985 WARN [c.c.u.AccountManagerImpl] (API-Job-Executo= r-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Failed to cleanup account Acct[83f26= 103-9d83-4e4e-8bcc-c282c2acb498-acc] due to > com.cloud.exception.InvalidParameterValueException: Can't delete snapshot= shot 6 due to it is in Allocated Status > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshot(XenserverSnapshotStrategy.java:213) > at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotD= irsForAccount(SnapshotManagerImpl.java:623) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source) > at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManage= rImpl.java:706) > at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManager= Impl.java:665) > at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountMan= agerImpl.java:1444) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at org.apache.cloudstack.network.contrail.management.EventUtils$E= ventInterceptor.invoke(EventUtils.java:106) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:161) > at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInter= ceptor.java:51) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:161) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy103.deleteUserAccount(Unknown Source) > at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccou= nt(RegionManagerImpl.java:187) > at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccou= nt(RegionServiceImpl.java:121) > at org.apache.cloudstack.api.command.admin.account.DeleteAccountC= md.execute(DeleteAccountCmd.java:104) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatch= er.java:108) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.runInContext(AsyncJobManagerImpl.java:513) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.run(AsyncJobManagerImpl.java:470) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-10-28 15:16:45,008 INFO [c.c.u.AccountManagerImpl] (API-Job-Executo= r-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Cleanup for account 4 is needed. > 2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bc= b44ca9) =3D=3D=3DSTART=3D=3D=3D 10.252.193.8 -- GET command=3DqueryAsyncJ= obResult&jobId=3D2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=3Djson&sessi= onkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=3D1414470673593 > 2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bc= b44ca9 ctx-82b3efc5) =3D=3D=3DEND=3D=3D=3D 10.252.193.8 -- GET command=3D= queryAsyncJobResult&jobId=3D2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response= =3Djson&sessionkey=3DEpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=3D1414470673593 > 2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Complete async job-53, jobSta= tus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.S= uccessResponse/null/{"success":true} > 2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Publish async job-53 complete= on message bus > 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs related to job- = 53 > 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Update db status for job- 53 > 2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs joined with job-= 53 and disjoin all subjobs created from job- 53 > 2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-= Executor-30:ctx-0b9151e2 job-53) Done executing org.apache.cloudstack.api.c= ommand.admin.account.DeleteAccountCmd for job-53 > 2014-10-28 15:16:45,248 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Exec= utor-30:ctx-0b9151e2 job-53) Remove job-53 from job monitoring > Even the account cleanup thread could not clean any of the remained artif= acts like vms,networks -- This message was sent by Atlassian JIRA (v6.3.4#6332)