cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: Orphaned VRouters
Date Tue, 24 Jul 2012 19:08:19 GMT
Please Zip it, place it on some publicly available web host and send the
link.

Thank you,
Alena.

On 7/24/12 11:59 AM, "Diego Spinola Castro" <spinolacastro@gmail.com>
wrote:

>What about management log. its 10mb , how do i send to you?
>
>mysql> select * from op_networks;
>+-----+-----------------+------------+----+--------------+
>| id  | mac_address_seq | nics_count | gc | check_for_gc |
>+-----+-----------------+------------+----+--------------+
>| 200 |            2069 |          0 |  0 |            0 |
>| 201 |               1 |          0 |  0 |            0 |
>| 202 |            1062 |          0 |  0 |            0 |
>| 203 |               1 |          0 |  0 |            0 |
>| 204 |             224 |          5 |  1 |            1 |
>| 205 |              13 |          0 |  1 |            0 |
>| 206 |               8 |          3 |  1 |            1 |
>| 207 |              12 |          0 |  1 |            0 |
>| 208 |               7 |          2 |  1 |            1 |
>| 209 |              10 |          0 |  1 |            0 |
>| 210 |               5 |          0 |  1 |            0 |
>| 211 |              10 |          1 |  1 |            1 |
>| 212 |              14 |          0 |  1 |            0 |
>| 213 |               8 |          6 |  1 |            1 |
>| 214 |               5 |          0 |  1 |            0 |
>| 215 |               4 |          0 |  1 |            0 |
>| 216 |               3 |          0 |  1 |            0 |
>| 217 |               3 |          0 |  1 |            0 |
>| 218 |               4 |          3 |  1 |            1 |
>| 219 |               3 |          0 |  1 |            0 |
>| 220 |              10 |          5 |  1 |            1 |
>| 221 |               6 |          6 |  1 |            1 |
>| 222 |              16 |          5 |  1 |            1 |
>| 223 |              14 |          1 |  1 |            1 |
>| 224 |              30 |          1 |  1 |            1 |
>| 225 |               3 |          0 |  1 |            0 |
>| 226 |               8 |         11 |  1 |            1 |
>| 227 |               3 |          0 |  1 |            0 |
>| 228 |               3 |          2 |  1 |            1 |
>| 229 |               4 |          3 |  1 |            1 |
>| 230 |               3 |          0 |  1 |            0 |
>| 231 |               6 |          7 |  1 |            1 |
>| 232 |              26 |         24 |  1 |            1 |
>| 233 |               4 |          2 |  1 |            1 |
>| 234 |               7 |          9 |  1 |            1 |
>| 235 |               3 |          0 |  1 |            0 |
>| 236 |               3 |          2 |  1 |            1 |
>| 237 |               4 |          0 |  1 |            0 |
>| 238 |               9 |          7 |  1 |            1 |
>| 240 |               4 |          3 |  1 |            1 |
>| 241 |               5 |          3 |  1 |            1 |
>| 242 |               3 |          3 |  1 |            1 |
>| 243 |              30 |          4 |  1 |            1 |
>| 244 |               3 |          0 |  1 |            0 |
>| 245 |               4 |          3 |  1 |            1 |
>| 246 |               4 |          3 |  1 |            1 |
>| 247 |               4 |          3 |  1 |            1 |
>| 248 |               3 |          2 |  1 |            1 |
>| 249 |               3 |          2 |  1 |            1 |
>| 250 |               3 |          3 |  1 |            1 |
>| 251 |               7 |          0 |  1 |            0 |
>| 252 |               4 |          6 |  1 |            1 |
>| 253 |               3 |          2 |  1 |            1 |
>| 254 |              13 |          6 |  1 |            1 |
>| 255 |               7 |          3 |  1 |            1 |
>| 256 |               3 |          1 |  1 |            1 |
>| 257 |               6 |          4 |  1 |            1 |
>| 258 |               1 |          6 |  1 |            1 |
>| 259 |               4 |          3 |  1 |            1 |
>| 260 |               5 |          2 |  1 |            1 |
>| 261 |               3 |          3 |  1 |            1 |
>| 262 |               5 |          0 |  1 |            0 |
>| 263 |               5 |          1 |  1 |            1 |
>| 264 |               3 |          0 |  1 |            0 |
>| 265 |               3 |          1 |  1 |            1 |
>| 266 |               4 |          1 |  1 |            1 |
>| 267 |               3 |          1 |  1 |            1 |
>| 268 |               6 |          2 |  1 |            1 |
>| 269 |               3 |          1 |  1 |            1 |
>| 270 |               4 |          0 |  1 |            0 |
>| 271 |               3 |          1 |  1 |            1 |
>| 272 |               3 |          1 |  1 |            1 |
>| 273 |               3 |          1 |  1 |            1 |
>| 274 |              10 |          2 |  1 |            1 |
>| 275 |               3 |          1 |  1 |            1 |
>+-----+-----------------+------------+----+--------------+
>
>mysql> select * from op_networks where id=236;
>+-----+-----------------+------------+----+--------------+
>| id  | mac_address_seq | nics_count | gc | check_for_gc |
>+-----+-----------------+------------+----+--------------+
>| 236 |               3 |          2 |  1 |            1 |
>+-----+-----------------+------------+----+--------------+
>
>mysql> select * from vm_instance where id in (select instance_id from nics
>where removed is null and network_id=236)\G
>*************************** 1. row ***************************
>                 id: 348
>               name: r-348-VM
>      instance_name: r-348-VM
>              state: Stopped
>     vm_template_id: 8
>        guest_os_id: 15
>private_mac_address: 02:00:13:63:03:ed
> private_ip_address: 10.16.140.70
>    private_netmask: NULL
>             pod_id: 1
>     data_center_id: 1
>            host_id: NULL
>       last_host_id: 2
>           proxy_id: 305
>  proxy_assign_time: 2012-03-17 04:43:35
>       vnc_password: d3cc123795551907
>         ha_enabled: 1
>      limit_cpu_use: 1
>       update_count: 202
>        update_time: 2012-07-17 15:54:54
>            created: 2012-02-16 16:37:19
>            removed: NULL
>               type: DomainRouter
>            vm_type: DomainRouter
>         account_id: 43
>          domain_id: 1
>service_offering_id: 7
>     reservation_id: 147062dc-3596-4799-a84e-667784810067
>    hypervisor_type: VMware
>1 row in set (0.00 sec)
>
>
>2012/7/24 Alena Prokharchyk <Alena.Prokharchyk@citrix.com>
>
>> Error message:
>>
>> "Unable to remove the network id=236 as it has active Nics".
>>
>>
>> Indicates that the User vm(s) (not router) fail to expunge during the
>> account deletion, therefore the corresponding network failed to be
>> removed. The account will be marked with Removed=someDate +
>> cleanup_needed=1, and the recurring thread (runs every
>> account.clenaup.interval defined in global config) will try to cleanup
>>the
>> account again and again, until it succeeds.
>>
>>
>>
>> To find out if this is the root cause, following is needed:
>>
>> * management server log
>> * content of the op_networks table, the active nic info is stored there.
>> Nics should be counted for non-expunged user vms only (select * from
>> op_networks where id=236)
>> * information about the user vms belonging to network id=236 (select *
>> from vm_instance where id in (select instance_id from nics where removed
>> is null and network_id=236))
>>
>> -Alena.
>>
>> On 7/24/12 11:15 AM, "Edison Su" <Edison.su@citrix.com> wrote:
>>
>> >Sounds like a bug, the VR is not got destroyed during deleteAccount.
>> >Could you help to fire a bug, and attach the whole mgt server log?
>> >
>> >> -----Original Message-----
>> >> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
>> >> Sent: Tuesday, July 24, 2012 10:03 AM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: Orphaned VRouters
>> >>
>> >> Hi, i found some orphan vrouters and don't know why they still
>>running
>> >> even
>> >> there's no account anymore (i deleted account).
>> >>
>> >> Bellow management.log
>> >>
>> >>
>> >> 2012-07-19 12:39:06,437 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> (http-12449-14:null) submit async job-6598, details: AsyncJobVO
>> >> {id:6598,
>> >> userId: 32, accountId: 2, sessionKey: null, instanceType: None,
>> >> instanceId:
>> >> null, cmd: com.cloud.api.commands.DeleteAccountCmd, cmdOriginator:
>>null,
>> >> cmdInfo:
>> >>
>>{"id":"43","response":"json","sessionkey":"7uBHMfOyDdSLdtSjPB9K0FcVloQ\
>> >>
>>u003d","ctxUserId":"32","_":"1342707903173","ctxAccountId":"2","ctxStar
>> >> tEventId":"35670"},
>> >> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
>> >> processStatus: 0, resultCode: 0, result: null, initMsid:
>>144350996536,
>> >> completeMsid: null, lastUpdated: null, lastPolled: null, created:
>>null}
>> >> 2012-07-19 12:39:06,438 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> (Job-Executor-25:job-6598) Executing
>> >> com.cloud.api.commands.DeleteAccountCmd for job-6598
>> >> 2012-07-19 12:39:06,443 DEBUG [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Access to Acct[43-300012_CUBOCC] granted
>>to
>> >> Acct[2-admin] by DomainChecker
>> >> 2012-07-19 12:39:06,460 DEBUG [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Remove account 43
>> >> 2012-07-19 12:39:06,514 DEBUG [agent.transport.Request]
>> >> (Job-Executor-25:job-6598) Seq 20-573703176: Sending  { Cmd , MgmtId:
>> >> 144350996536, via: 20, Ver: v1, Flags: 100011,
>> >>
>>[{"DeleteSnapshotsDirCommand":{"primaryStoragePoolNameLabel":"016b139d-
>> >> a43e-32de-a330-e57a91388695","secondaryStoragePoolURL":"nfs://
>> >>
>>10.16.140.11/export","dcId":1,"accountId":43,"volumeId":467,"volumePath
>> >> ":"2c880c4443a44672971b964d0cfe725c","wait":0}}]
>> >> }
>> >> 2012-07-19 12:39:09,334 DEBUG [agent.transport.Request]
>> >> (Job-Executor-25:job-6598) Seq 20-573703176: Received:  { Ans: ,
>>MgmtId:
>> >> 144350996536, via: 20, Ver: v1, Flags: 10, { Answer } }
>> >> 2012-07-19 12:39:09,343 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details null
>> >> 2012-07-19 12:39:09,343 DEBUG [storage.snapshot.SnapshotManagerImpl]
>> >> (Job-Executor-25:job-6598) Deleted snapshotsDir for volume: 467 under
>> >> account: 43
>> >> 2012-07-19 12:39:09,345 DEBUG [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Successfully deleted snapshots directories
>> >> for
>> >> all volumes under account 43 across all zones
>> >> 2012-07-19 12:39:09,348 DEBUG [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Destroying # of vms (accountId=43): 1
>> >> 2012-07-19 12:39:09,348 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (Job-Executor-25:job-6598) Stopped called on VM[User|i-43-347-VM] but
>> >> the
>> >> state is Destroyed
>> >> 2012-07-19 12:39:09,355 DEBUG [cloud.capacity.CapacityManagerImpl]
>> >> (Job-Executor-25:job-6598) VM state transitted from :Destroyed to
>> >> Expunging
>> >> with event: ExpungeOperationvm's original host id: 4 new host id:
>>null
>> >> host
>> >> id before state transition: null
>> >> 2012-07-19 12:39:09,355 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (Job-Executor-25:job-6598) Destroying vm VM[User|i-43-347-VM]
>> >> 2012-07-19 12:39:09,355 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Cleaning network for vm: 347
>> >> 2012-07-19 12:39:09,369 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (Job-Executor-25:job-6598) Cleaning storage for vm: 347
>> >> 2012-07-19 12:39:09,401 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (Job-Executor-25:job-6598) Detaching Vol[467|vm=347|DATADISK]
>> >> 2012-07-19 12:39:09,402 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (Job-Executor-25:job-6598) Detaching Vol[488|vm=347|DATADISK]
>> >> 2012-07-19 12:39:09,404 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (Job-Executor-25:job-6598) Expunging Vol[392|vm=347|ROOT]
>> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
>> >> (Job-Executor-25:job-6598) Seq 3-115876736: Sending  { Cmd , MgmtId:
>> >> 144350996536, via: 3, Ver: v1, Flags: 100111,
>> >> [{"storage.DestroyCommand":{"vmName":"i-43-347-
>> >> VM","volume":{"id":392,"name":"ROOT-
>> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
>> >>
>>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
>> >> PoolUuid":"30b0d978-3291-3ce3-a06f-
>> >> eea3ac2f4f51","deviceId":0},"wait":0}}]
>> >> }
>> >> 2012-07-19 12:39:09,407 DEBUG [agent.transport.Request]
>> >> (Job-Executor-25:job-6598) Seq 3-115876736: Executing:  { Cmd ,
>>MgmtId:
>> >> 144350996536, via: 3, Ver: v1, Flags: 100111,
>> >> [{"storage.DestroyCommand":{"vmName":"i-43-347-
>> >> VM","volume":{"id":392,"name":"ROOT-
>> >> 347","mountPoint":"/TELIUMSPO/SATA_R5_VOL3","path":"ROOT-
>> >>
>>347","size":53687091200,"type":"ROOT","storagePoolType":"VMFS","storage
>> >> PoolUuid":"30b0d978-3291-3ce3-a06f-
>> >> eea3ac2f4f51","deviceId":0},"wait":0}}]
>> >> }
>> >> 2012-07-19 12:39:27,879 DEBUG [agent.transport.Request]
>> >> (Job-Executor-25:job-6598) Seq 3-115876736: Received:  { Ans: ,
>>MgmtId:
>> >> 144350996536, via: 3, Ver: v1, Flags: 110, { Answer } }
>> >> 2012-07-19 12:39:27,879 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (Job-Executor-25:job-6598) Cleanup succeeded. Details Success
>> >> 2012-07-19 12:39:27,890 DEBUG [cloud.storage.StorageManagerImpl]
>> >> (Job-Executor-25:job-6598) Volume successfully expunged from 206
>> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.VirtualMachineManagerImpl]
>> >> (Job-Executor-25:job-6598) Expunged VM[User|i-43-347-VM]
>> >> 2012-07-19 12:39:27,890 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (Job-Executor-25:job-6598) Starting cleaning up vm
>>VM[User|i-43-347-VM]
>> >> resources...
>> >> 2012-07-19 12:39:27,947 DEBUG [network.firewall.FirewallManagerImpl]
>> >> (Job-Executor-25:job-6598) No firewall rules are found for vm id=347
>> >> 2012-07-19 12:39:27,947 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (Job-Executor-25:job-6598) Firewall rules are removed successfully
>>as a
>> >> part of vm id=347 expunge
>> >> 2012-07-19 12:39:27,995 DEBUG [network.rules.RulesManagerImpl]
>> >> (Job-Executor-25:job-6598) Applying port forwarding rules for ip
>> >> address
>> >> id=58 as a part of vm expunge
>> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> >> handled
>> >> by ExternalFirewall
>> >> 2012-07-19 12:39:28,001 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> >> handled
>> >> by ExternalLoadBalancer
>> >> 2012-07-19 12:39:28,003 DEBUG
>> >> [network.router.VirtualNetworkApplianceManagerImpl]
>> >> (Job-Executor-25:job-6598) Router r-348-VM is in Stopped, so not
>> >> sending
>> >> apply firewall rules commands to the backend
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were
>>handled
>> >> by
>> >> DomainRouter
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> >> handled
>> >> by Dhcp
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were
>>handled
>> >> by
>> >> Ovs
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were
>>handled
>> >> by
>> >> ExternalDhcp
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were
>>handled
>> >> by
>> >> BareMetal
>> >> 2012-07-19 12:39:28,003 DEBUG
>> >> [network.element.ElasticLoadBalancerElement]
>> >> (Job-Executor-25:job-6598) Not handling network with guest Type
>> >> Virtual
>> >> and traffic type Guest
>> >> 2012-07-19 12:39:28,003 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Network Rules for network 236 were  not
>> >> handled
>> >> by ElasticLoadBalancer
>> >> 2012-07-19 12:39:28,034 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (Job-Executor-25:job-6598) Port forwarding rules are removed
>> >> successfully
>> >> as a part of vm id=347 expunge
>> >> 2012-07-19 12:39:28,045 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (Job-Executor-25:job-6598) Removed vm id=347 from all load balancers
>>as
>> >> a
>> >> part of expunge process
>> >> 2012-07-19 12:39:28,046 DEBUG [cloud.vm.UserVmManagerImpl]
>> >> (Job-Executor-25:job-6598) Successfully cleaned up vm
>>VM[User|i-43-347-
>> >> VM]
>> >> resources as a part of expunge process
>> >> 2012-07-19 12:39:28,108 INFO  [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) deleteAccount: Deleted 1 network groups
>>for
>> >> account 43
>> >> 2012-07-19 12:39:28,108 DEBUG [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Deleting networks for account 43
>> >> 2012-07-19 12:39:28,111 DEBUG [cloud.network.NetworkManagerImpl]
>> >> (Job-Executor-25:job-6598) Unable to remove the network id=236 as it
>> >> has
>> >> active Nics.
>> >> 2012-07-19 12:39:28,111 WARN  [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Unable to destroy network
>>Ntwk[236|Guest|6]
>> >> as a
>> >> part of account id=43 cleanup.
>> >> 2012-07-19 12:39:28,114 INFO  [cloud.user.AccountManagerImpl]
>> >> (Job-Executor-25:job-6598) Cleanup for account 43 is needed.
>> >> 2012-07-19 12:39:28,126 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> (Job-Executor-25:job-6598) Complete async job-6598, jobStatus: 1,
>> >> resultCode: 0, result:
>>com.cloud.api.response.SuccessResponse@49ce1227
>> >> 2012-07-19 12:39:28,129 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> (Job-Executor-25:job-6598) Done executing
>> >> com.cloud.api.commands.DeleteAccountCmd for job-6598
>> >> 2012-07-19 12:39:36,657 DEBUG [cloud.async.AsyncJobManagerImpl]
>> >> (http-12449-14:null) Async job-6598 completed
>> >>
>> >>
>> >> PS: all vms, snapshots and volumes was vanished after account
>>deleted.
>> >> just
>> >> vrouter wasn't.
>> >
>>
>>
>>
>



Mime
View raw message