incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: Orphaned VRouters
Date Tue, 24 Jul 2012 18:42:10 GMT
Error message:

"Unable to remove the network id=236 as it has active Nics".


Indicates that the User vm(s) (not router) fail to expunge during the
account deletion, therefore the corresponding network failed to be
removed. The account will be marked with Removed=someDate +
cleanup_needed=1, and the recurring thread (runs every
account.clenaup.interval defined in global config) will try to cleanup the
account again and again, until it succeeds.



To find out if this is the root cause, following is needed:

* management server log
* content of the op_networks table, the active nic info is stored there.
Nics should be counted for non-expunged user vms only (select * from
op_networks where id=236)
* information about the user vms belonging to network id=236 (select *
from vm_instance where id in (select instance_id from nics where removed
is null and network_id=236))

-Alena.

On 7/24/12 11:15 AM, "Edison Su" <Edison.su@citrix.com> wrote:

>Sounds like a bug, the VR is not got destroyed during deleteAccount.
>Could you help to fire a bug, and attach the whole mgt server log?
>
>> -----Original Message-----
>> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
>> Sent: Tuesday, July 24, 2012 10:03 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Orphaned VRouters
>> 
>> Hi, i found some orphan vrouters and don't know why they still running
>> even
>> there's no account anymore (i deleted account).
>> 
>> Bellow management.log
>> 
>> 
>> 2012-07-19 12:39:06,437 DEBUG [cloud.async.AsyncJobManagerImpl]
>> (http-12449-14:null) submit async job-6598, details: AsyncJobVO
>> {id:6598,
>> userId: 32, accountId: 2, sessionKey: null, instanceType: None,
>> instanceId:
>> null, cmd: com.cloud.api.commands.DeleteAccountCmd, cmdOriginator: null,
>> cmdInfo:
>> {"id":"43","response":"json","sessionkey":"7uBHMfOyDdSLdtSjPB9K0FcVloQ\
>> u003d","ctxUserId":"32","_":"1342707903173","ctxAccountId":"2","ctxStar
>> tEventId":"35670"},
>> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
>> processStatus: 0, resultCode: 0, result: null, initMsid: 144350996536,
>> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
>> 2012-07-19 12:39:06,438 DEBUG [cloud.async.AsyncJobManagerImpl]
>> (Job-Executor-25:job-6598) Executing
>> com.cloud.api.commands.DeleteAccountCmd for job-6598
>> 2012-07-19 12:39:06,443 DEBUG [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Access to Acct[43-300012_CUBOCC] granted to
>> Acct[2-admin] by DomainChecker
>> 2012-07-19 12:39:06,460 DEBUG [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Remove account 43
>> 2012-07-19 12:39:06,514 DEBUG [agent.transport.Request]
>> (Job-Executor-25:job-6598) Seq 20-573703176: Sending  { Cmd , MgmtId:
>> 144350996536, via: 20, Ver: v1, Flags: 100011,
>> [{"DeleteSnapshotsDirCommand":{"primaryStoragePoolNameLabel":"016b139d-
>> a43e-32de-a330-e57a91388695","secondaryStoragePoolURL":"nfs://
>> 10.16.140.11/export","dcId":1,"accountId":43,"volumeId":467,"volumePath
>> ":"2c880c4443a44672971b964d0cfe725c","wait":0}}]
>> }
>> 2012-07-19 12:39:09,334 DEBUG [agent.transport.Request]
>> (Job-Executor-25:job-6598) Seq 20-573703176: Received:  { Ans: , MgmtId:
>> 144350996536, via: 20, Ver: v1, Flags: 10, { Answer } }
>> 2012-07-19 12:39:09,343 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-25:job-6598) Cleanup succeeded. Details null
>> 2012-07-19 12:39:09,343 DEBUG [storage.snapshot.SnapshotManagerImpl]
>> (Job-Executor-25:job-6598) Deleted snapshotsDir for volume: 467 under
>> account: 43
>> 2012-07-19 12:39:09,345 DEBUG [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Successfully deleted snapshots directories
>> for
>> all volumes under account 43 across all zones
>> 2012-07-19 12:39:09,348 DEBUG [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Destroying # of vms (accountId=43): 1
>> 2012-07-19 12:39:09,348 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-25:job-6598) Stopped called on VM[User|i-43-347-VM] but
>> the
>> state is Destroyed
>> 2012-07-19 12:39:09,355 DEBUG [cloud.capacity.CapacityManagerImpl]
>> (Job-Executor-25:job-6598) VM state transitted from :Destroyed to
>> Expunging
>> with event: ExpungeOperationvm's original host id: 4 new host id: null
>> host
>> id before state transition: null
>> 2012-07-19 12:39:09,355 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-25:job-6598) Destroying vm VM[User|i-43-347-VM]
>> 2012-07-19 12:39:09,355 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Cleaning network for vm: 347
>> 2012-07-19 12:39:09,369 DEBUG [cloud.storage.StorageManagerImpl]
>> (Job-Executor-25:job-6598) Cleaning storage for vm: 347
>> 2012-07-19 12:39:09,401 DEBUG [cloud.storage.StorageManagerImpl]
>> (Job-Executor-25:job-6598) Detaching Vol[467|vm=347|DATADISK]
>> 2012-07-19 12:39:09,402 DEBUG [cloud.storage.StorageManagerImpl]
>> (Job-Executor-25:job-6598) Detaching Vol[488|vm=347|DATADISK]
>> 2012-07-19 12:39:09,404 DEBUG [cloud.storage.StorageManagerImpl]
>> (Job-Executor-25:job-6598) Expunging Vol[392|vm=347|ROOT]
>> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
>> (Job-Executor-25:job-6598) Seq 3-115876736: Sending  { Cmd , MgmtId:
>> 144350996536, via: 3, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"i-43-347-
>> VM","volume":{"id":392,"name":"ROOT-
>> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
>> 347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
>> PoolUuid":"30b0d978-3291-3ce3-a06f-
>> eea3ac2f4f51","deviceId":0},"wait":0}}]
>> }
>> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
>> (Job-Executor-25:job-6598) Seq 3-115876736: Executing:  { Cmd , MgmtId:
>> 144350996536, via: 3, Ver: v1, Flags: 100111,
>> [{"storage.DestroyCommand":{"vmName":"i-43-347-
>> VM","volume":{"id":392,"name":"ROOT-
>> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
>> 347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
>> PoolUuid":"30b0d978-3291-3ce3-a06f-
>> eea3ac2f4f51","deviceId":0},"wait":0}}]
>> }
>> 2012-07-19 12:39:27,879 DEBUG [agent.transport.Request]
>> (Job-Executor-25:job-6598) Seq 3-115876736: Received:  { Ans: , MgmtId:
>> 144350996536, via: 3, Ver: v1, Flags: 110, { Answer } }
>> 2012-07-19 12:39:27,879 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-25:job-6598) Cleanup succeeded. Details Success
>> 2012-07-19 12:39:27,890 DEBUG [cloud.storage.StorageManagerImpl]
>> (Job-Executor-25:job-6598) Volume successfully expunged from 206
>> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-25:job-6598) Expunged VM[User|i-43-347-VM]
>> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.UserVmManagerImpl]
>> (Job-Executor-25:job-6598) Starting cleaning up vm VM[User|i-43-347-VM]
>> resources...
>> 2012-07-19 12:39:27,947 DEBUG [network.firewall.FirewallManagerImpl]
>> (Job-Executor-25:job-6598) No firewall rules are found for vm id=347
>> 2012-07-19 12:39:27,947 DEBUG [cloud.vm.UserVmManagerImpl]
>> (Job-Executor-25:job-6598) Firewall rules are removed successfully as a
>> part of vm id=347 expunge
>> 2012-07-19 12:39:27,995 DEBUG [network.rules.RulesManagerImpl]
>> (Job-Executor-25:job-6598) Applying port forwarding rules for ip
>> address
>> id=58 as a part of vm expunge
>> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> handled
>> by ExternalFirewall
>> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> handled
>> by ExternalLoadBalancer
>> 2012-07-19 12:39:28,003 DEBUG
>> [network.router.VirtualNetworkApplianceManagerImpl]
>> (Job-Executor-25:job-6598) Router r-348-VM is in Stopped, so not
>> sending
>> apply firewall rules commands to the backend
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  handled
>> by
>> DomainRouter
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> handled
>> by Dhcp
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  handled
>> by
>> Ovs
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  handled
>> by
>> ExternalDhcp
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  handled
>> by
>> BareMetal
>> 2012-07-19 12:39:28,003 DEBUG
>> [network.element.ElasticLoadBalancerElement]
>> (Job-Executor-25:job-6598) Not handling network with guest Type
>> Virtual
>> and traffic type Guest
>> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> handled
>> by ElasticLoadBalancer
>> 2012-07-19 12:39:28,034 DEBUG [cloud.vm.UserVmManagerImpl]
>> (Job-Executor-25:job-6598) Port forwarding rules are removed
>> successfully
>> as a part of vm id=347 expunge
>> 2012-07-19 12:39:28,045 DEBUG [cloud.vm.UserVmManagerImpl]
>> (Job-Executor-25:job-6598) Removed vm id=347 from all load balancers as
>> a
>> part of expunge process
>> 2012-07-19 12:39:28,046 DEBUG [cloud.vm.UserVmManagerImpl]
>> (Job-Executor-25:job-6598) Successfully cleaned up vm VM[User|i-43-347-
>> VM]
>> resources as a part of expunge process
>> 2012-07-19 12:39:28,108 INFO  [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) deleteAccount: Deleted 1 network groups for
>> account 43
>> 2012-07-19 12:39:28,108 DEBUG [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Deleting networks for account 43
>> 2012-07-19 12:39:28,111 DEBUG [cloud.network.NetworkManagerImpl]
>> (Job-Executor-25:job-6598) Unable to remove the network id=236 as it
>> has
>> active Nics.
>> 2012-07-19 12:39:28,111 WARN  [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Unable to destroy network Ntwk[236|Guest|6]
>> as a
>> part of account id=43 cleanup.
>> 2012-07-19 12:39:28,114 INFO  [cloud.user.AccountManagerImpl]
>> (Job-Executor-25:job-6598) Cleanup for account 43 is needed.
>> 2012-07-19 12:39:28,126 DEBUG [cloud.async.AsyncJobManagerImpl]
>> (Job-Executor-25:job-6598) Complete async job-6598, jobStatus: 1,
>> resultCode: 0, result: com.cloud.api.response.SuccessResponse@49ce1227
>> 2012-07-19 12:39:28,129 DEBUG [cloud.async.AsyncJobManagerImpl]
>> (Job-Executor-25:job-6598) Done executing
>> com.cloud.api.commands.DeleteAccountCmd for job-6598
>> 2012-07-19 12:39:36,657 DEBUG [cloud.async.AsyncJobManagerImpl]
>> (http-12449-14:null) Async job-6598 completed
>> 
>> 
>> PS: all vms, snapshots and volumes was vanished after account deleted.
>> just
>> vrouter wasn't.
>



Mime
View raw message