cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjeev N (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-7799) [Critical][Accounts] Account deletion failed to remove all the entities related to it in case of failure in deleting one entity
Date Tue, 28 Oct 2014 05:05:33 GMT
Sanjeev N created CLOUDSTACK-7799:
-------------------------------------

             Summary: [Critical][Accounts] Account deletion failed to remove all the entities
related to it in case of failure in deleting one entity
                 Key: CLOUDSTACK-7799
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7799
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Management Server
    Affects Versions: 4.5.0
         Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]# cloudstack-sccs
385c4f673dfbd1fd326e539625e2c06db4cdc27d

            Reporter: Sanjeev N
            Priority: Critical
             Fix For: 4.5.0


[Critical][Accounts] Account deletion failed to remove all the entities related to it in case
of failure in deleting one entity

Steps to Reproduce:
================
1.Bring up CS in advanced zone with one vmware cluster
2.Create a user account 
3.With the user account deploy few (4-5) vms 
4.Take snapshots on all of the vms root disks
5.Deploy another vm and simulate snapshot failure operation(In my case I tried snapshot operation
with quiesce option set to true) so that snapshot will be in "Allocated" state
6.Delete this user account

Expected Behavior:
==============
Whatever may the state of the snapshot account deletion shall clean all the objects related
that account

Actual Behavior:
=============
Account deletion started deleting snapshots. Since one of the snapshots is in Allocated state
it failed to delete that snapshot and the Account clean up job ended there. So vms and networks
related to the accounts are still there but the account was deleted.

Here is the log snippet:
=================
2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72) ===START===
 10.252.193.8 -- GET  command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-5:ctx-1c16bc72
ctx-51140d28) submit async job-53, details: AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType:
Account, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd,
cmdInfo: {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
6637838401571, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72 ctx-51140d28)
===END===  10.252.193.8 -- GET  command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,996 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-30:ctx-0b9151e2
job-53) Add job-53 into job monitoring
2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53) Executing AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo: {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
6637838401571, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Removed account 4
2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292948: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/5","wait":0}}]
}
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 4-4574812796478292948:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292948: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 5 under account: 4
2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) Seq 4-4574812796478292948:
No more commands found
2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 1
2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Snapshot: 1 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292949: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89-d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":2147483648,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA","name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (AgentManager-Handler-17:null) Seq 4-4574812796478292949:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn't exist","wait":0}}] }
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292949: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
10, { Answer } }
2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292950: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/7","wait":0}}]
}
2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca) ===START===
 10.252.193.8 -- GET  command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca ctx-9e36e958)
===END===  10.252.193.8 -- GET  command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 4-4574812796478292950:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292950: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 7 under account: 4
2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) Seq 4-4574812796478292950:
No more commands found
2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 2
2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Snapshot: 2 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292951: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a-869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":2147483648,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA","name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request] (AgentManager-Handler-16:null) Seq 4-4574812796478292951:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory ecb858aa-1656-4594-996a-869f1695bce2 doesn't exist","wait":0}}] }
2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292951: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
10, { Answer } }
2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292952: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/8","wait":0}}]
}
2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request] (AgentManager-Handler-19:null) Seq 4-4574812796478292952:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292952: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 8 under account: 4
2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-19:null) Seq 4-4574812796478292952:
No more commands found
2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 3
2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Snapshot: 3 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292953: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b-cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebff64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":2147483648,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA","name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null) Seq 4-4574812796478292953:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory 657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn't exist","wait":0}}] }
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292953: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
10, { Answer } }
2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292954: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/9","wait":0}}]
}
2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-6ecc0a29) AutoScaling
Monitor is running...
2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 4-4574812796478292954:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292954: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 9 under account: 4
2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) Seq 4-4574812796478292954:
No more commands found
2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 4
2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Snapshot: 4 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292955: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572-09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":2147483648,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA","name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) Seq 4-4574812796478292955:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory e7d15b83-06be-4596-9572-09478415e776 doesn't exist","wait":0}}] }
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292955: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
10, { Answer } }
2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292956: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/10","wait":0}}]
}
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) Seq 4-4574812796478292956:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) Seq 4-4574812796478292956:
No more commands found
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292956: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 10 under account: 4
2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 5
2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Snapshot: 5 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292957: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":5368709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_20141027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) Seq 4-4574812796478292957:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory 1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't exist","wait":0}}] }
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292957: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
10, { Answer } }
2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292958: Sending  { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/11","wait":0}}]
}
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null) Seq 4-4574812796478292958:
Processing:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot
directory 11 doesn't exist","wait":0}}] }
2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) Seq 4-4574812796478292958:
No more commands found
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24)
Seq 4-4574812796478292958: Received:  { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags:
110, { Answer } }
2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Deleted all snapshots for volume: 11 under account: 4
2014-10-28 15:16:44,985 WARN  [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Failed to cleanup account Acct[83f26103-9d83-4e4e-8bcc-c282c2acb498-acc]
due to
com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 6 due to it
is in Allocated Status
        at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source)
        at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:706)
        at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:665)
        at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1444)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy103.deleteUserAccount(Unknown Source)
        at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:187)
        at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:121)
        at org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:104)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2014-10-28 15:16:45,008 INFO  [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Cleanup for account 4 is needed.
2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bcb44ca9) ===START===
 10.252.193.8 -- GET  command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bcb44ca9 ctx-82b3efc5)
===END===  10.252.193.8 -- GET  command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Complete async job-53, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Publish async job-53 complete on message bus
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Wake up jobs related to job- 53
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Update db status for job- 53
2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53 ctx-3bc74f24) Wake up jobs joined with job- 53 and disjoin all subjobs created from
job- 53
2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2
job-53) Done executing org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for
job-53
2014-10-28 15:16:45,248 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-30:ctx-0b9151e2
job-53) Remove job-53 from job monitoring

Even the account cleanup thread could not clean any of the remained artifacts like vms,networks




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message