Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9984B10C2C for ; Sat, 27 Jul 2013 09:17:54 +0000 (UTC) Received: (qmail 10398 invoked by uid 500); 27 Jul 2013 09:17:53 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 10367 invoked by uid 500); 27 Jul 2013 09:17:51 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 10355 invoked by uid 500); 27 Jul 2013 09:17:49 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 10345 invoked by uid 99); 27 Jul 2013 09:17:48 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jul 2013 09:17:48 +0000 Date: Sat, 27 Jul 2013 09:17:48 +0000 (UTC) From: "Likitha Shetty (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Assigned] (CLOUDSTACK-3876) Runtime Exceptions while deleting account which has snapshots created from volume MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-3876?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Likitha Shetty reassigned CLOUDSTACK-3876: ------------------------------------------ Assignee: Likitha Shetty =20 > Runtime Exceptions while deleting account which has snapshots created fro= m volume > -------------------------------------------------------------------------= -------- > > Key: CLOUDSTACK-3876 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-387= 6 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Storage Controller > Affects Versions: 4.2.0 > Reporter: Sailaja Mada > Assignee: Likitha Shetty > Fix For: 4.2.0 > > > Steps: > 1. Configure AdvZone with Xenserver > 2. Create new account=20 > 3. Deploy VM > 4. Add new volume and attach to this instance > 5. Create snapshots to this volume ( Total4 ) > 6. Delete the account=20 > 7. Account got deleted with no issues but there were Runtime exceptions i= n the log while deleting the snapshots of DATA volume. > 2013-07-27 14:24:55,363 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:nul= l) =3D=3D=3DSTART=3D=3D=3D 10.104.255.13 -- GET command=3DdeleteAccount&r= esponse=3Djson&sessionkey=3DC%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=3D7cf7f1c8-= b1f2-4754-9049-10506322e037&_=3D1374915066313 > 2013-07-27 14:24:55,398 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina= -exec-2:null) submit async job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364b= a ], details: AsyncJobVO {id:45, userId: 2, accountId: 2, sessionKey: null,= instanceType: Account, instanceId: null, cmd: org.apache.cloudstack.api.co= mmand.admin.account.DeleteAccountCmd, cmdOriginator: null, cmdInfo: {"respo= nse":"json","id":"7cf7f1c8-b1f2-4754-9049-10506322e037","sessionkey":"C/rzj= dA6B1wbWrgmfiasGDl6A6Q\u003d","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"= 2","httpmethod":"GET","_":"1374915066313","ctxAccountId":"2","ctxStartEvent= Id":"163"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: = 0, processStatus: 0, resultCode: 0, result: null, initMsid: 226870599129537= , completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-07-27 14:24:55,401 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:nul= l) =3D=3D=3DEND=3D=3D=3D 10.104.255.13 -- GET command=3DdeleteAccount&res= ponse=3Djson&sessionkey=3DC%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=3D7cf7f1c8-b1= f2-4754-9049-10506322e037&_=3D1374915066313 > 2013-07-27 14:24:55,403 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exec= utor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Executing org.= apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-45 =3D= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ] > 2013-07-27 14:24:55,434 DEBUG [cloud.user.AccountManagerImpl] (Job-Execut= or-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Removed account = 4 > 2013-07-27 14:24:55,459 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774981: Send= ing { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100111, [{"co= m.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api= .to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_rol= e":"Image"}},"directory":"snapshots/4/16","wait":0}}] } > 2013-07-27 14:24:55,551 DEBUG [agent.transport.Request] (AgentManager-Han= dler-2:null) Seq 5-1160774981: Processing: { Ans: , MgmtId: 22687059912953= 7, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tr= ue,"wait":0}}] } > 2013-07-27 14:24:55,551 DEBUG [agent.manager.AgentAttache] (AgentManager-= Handler-2:null) Seq 5-1160774981: No more commands found > 2013-07-27 14:24:55,551 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774981: Rece= ived: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 110, { Ans= wer } } > 2013-07-27 14:24:55,551 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job= -Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted a= ll snapshots for volume: 16 under account: 4 > 2013-07-27 14:24:55,567 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) del= ete snapshot chain for snapshot: 4 > 2013-07-27 14:24:55,569 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Sna= pshot: 4 doesn't have children, so it's ok to delete it and its parents > 2013-07-27 14:24:55,616 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774982: Send= ing { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100011, [{"or= g.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.clo= udstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/16/6671a6a2-565f-= 47ec-96cf-f804665c600d","volume":{"uuid":"bb9c0beb-ba86-4d03-9cbe-66dad892c= d57","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.Pri= maryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":200,"p= oolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhin= av/xen-pri1","port":2049}},"name":"ROOT-11","size":21474836480,"path":"12bb= 3e53-db3b-4d47-ab01-a4d8bbd5bbc3","volumeId":16,"vmName":"i-4-11-VM","accou= ntId":4,"format":"VHD","id":16},"dataStore":{"com.cloud.agent.api.to.NfsTO"= :{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"= }},"vmName":"i-4-11-VM","name":"instance1_ROOT-11_20130727072147","hypervis= orType":"XenServer","id":4}},"wait":0}}] } > 2013-07-27 14:24:55,641 DEBUG [agent.transport.Request] (AgentManager-Han= dler-4:null) Seq 5-1160774982: Processing: { Ans: , MgmtId: 22687059912953= 7, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":fal= se,"details":"snapshot directory 16 doesn't exist","wait":0}}] } > 2013-07-27 14:24:55,642 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774982: Rece= ived: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 10, { Answ= er } } > 2013-07-27 14:24:55,642 DEBUG [storage.snapshot.SnapshotServiceImpl] (Job= -Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete sn= apshot failedsnapshot directory 16 doesn't exist > 2013-07-27 14:24:55,673 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Fai= led to delete snapshot: > com.cloud.utils.exception.CloudRuntimeException: snapshot directory 16 do= esn't exist > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.del= eteSnapshot(SnapshotServiceImpl.java:375) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshot(XenserverSnapshotStrategy.java:187) > at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotD= irsForAccount(SnapshotManagerImpl.java:689) > at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManage= rImpl.java:581) > at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManager= Impl.java:544) > at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountMan= agerImpl.java:1273) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$= InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12= 5) > at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccou= nt(RegionManagerImpl.java:177) > at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccou= nt(RegionServiceImpl.java:118) > at org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.exec= ute(DeleteAccountCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.= java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2013-07-27 14:24:55,736 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774983: Send= ing { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100111, [{"co= m.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api= .to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_rol= e":"Image"}},"directory":"snapshots/4/18","wait":0}}] } > 2013-07-27 14:24:55,866 DEBUG [agent.transport.Request] (AgentManager-Han= dler-3:null) Seq 5-1160774983: Processing: { Ans: , MgmtId: 22687059912953= 7, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":tr= ue,"wait":0}}] } > 2013-07-27 14:24:55,866 DEBUG [agent.manager.AgentAttache] (AgentManager-= Handler-3:null) Seq 5-1160774983: No more commands found > 2013-07-27 14:24:55,866 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774983: Rece= ived: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 110, { Ans= wer } } > 2013-07-27 14:24:55,866 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job= -Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted a= ll snapshots for volume: 18 under account: 4 > 2013-07-27 14:24:55,882 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) del= ete snapshot chain for snapshot: 8 > 2013-07-27 14:24:55,885 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the= snapshot has child, can't delete it on the storage > 2013-07-27 14:24:55,944 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) del= ete snapshot chain for snapshot: 9 > 2013-07-27 14:24:55,947 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the= snapshot has child, can't delete it on the storage > 2013-07-27 14:24:56,011 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) del= ete snapshot chain for snapshot: 10 > 2013-07-27 14:24:56,014 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the= snapshot has child, can't delete it on the storage > 2013-07-27 14:24:56,075 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) del= ete snapshot chain for snapshot: 11 > 2013-07-27 14:24:56,077 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Sna= pshot: 11 doesn't have children, so it's ok to delete it and its parents > 2013-07-27 14:24:56,129 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774984: Send= ing { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100011, [{"or= g.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.clo= udstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/18/c44a1894-0142-= 444a-9c75-868358893a6a","volume":{"uuid":"f4e0e5fa-1a98-4dbb-b7b4-10e4ddf28= 465","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to= .PrimaryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":20= 0,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/a= bhinav/xen-pri1","port":2049}},"name":"Volumenew1","size":5368709120,"path"= :"a06f2cde-31f2-41e3-bdaa-82d224996856","volumeId":18,"vmName":"i-4-11-VM",= "accountId":4,"format":"VHD","id":18},"parentSnapshotPath":"snapshots/4/18/= c44a1894-0142-444a-9c75-868358893a6a","dataStore":{"com.cloud.agent.api.to.= NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"= Image"}},"vmName":"i-4-11-VM","name":"instance1_Volumenew1_20130727085422",= "hypervisorType":"XenServer","id":11}},"wait":0}}] } > 2013-07-27 14:24:56,174 DEBUG [agent.transport.Request] (AgentManager-Han= dler-7:null) Seq 5-1160774984: Processing: { Ans: , MgmtId: 22687059912953= 7, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":fal= se,"details":"snapshot directory 18 doesn't exist","wait":0}}] } > 2013-07-27 14:24:56,175 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774984: Rece= ived: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 10, { Answ= er } } > 2013-07-27 14:24:56,175 DEBUG [storage.snapshot.SnapshotServiceImpl] (Job= -Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete sn= apshot failedsnapshot directory 18 doesn't exist > 2013-07-27 14:24:56,192 DEBUG [storage.snapshot.XenserverSnapshotStrategy= ] (Job-Executor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Fai= led to delete snapshot: > com.cloud.utils.exception.CloudRuntimeException: snapshot directory 18 do= esn't exist > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.del= eteSnapshot(SnapshotServiceImpl.java:375) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.deleteSnapshot(XenserverSnapshotStrategy.java:187) > at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotD= irsForAccount(SnapshotManagerImpl.java:689) > at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManage= rImpl.java:581) > at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManager= Impl.java:544) > at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountMan= agerImpl.java:1273) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$= InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12= 5) > at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccou= nt(RegionManagerImpl.java:177) > at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccou= nt(RegionServiceImpl.java:118) > at org.apache.cloudstack.api.command.admin.account.DeleteAccountC= md.execute(DeleteAccountCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.= java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2013-07-27 14:24:56,234 DEBUG [cloud.user.AccountManagerImpl] (Job-Execut= or-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Successfully del= eted snapshots directories for all volumes under account 4 across all zones > 2013-07-27 14:24:56,238 DEBUG [cloud.user.AccountManagerImpl] (Job-Execut= or-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Expunging # of v= ms (accountId=3D4): 1 > 2013-07-27 14:24:56,261 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-E= xecutor-34:job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) VM state tr= ansitted from :Running to Stopping with event: StopRequestedvm's original h= ost id: 1 new host id: 1 host id before state transition: 1 > 2013-07-27 14:24:56,265 DEBUG [agent.transport.Request] (Job-Executor-34:= job-45 =3D [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 1-1592922337: Send= ing { Cmd , MgmtId: 226870599129537, via: 1, Ver: v1, Flags: 100011, [{"co= m.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"= vmName":"i-4-11-VM","wait":0}}] } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira