incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: Can not delete network
Date Wed, 13 Jun 2012 21:01:20 GMT
I see 2 different problems in this log fragment:

1)job-92 -  i-2-415-VM failed to start because the network vlan was
misconfigured. The number format exception most likely indicates the vlan
for the network was set incorrectly. Need entire log to confirm.

2) job-94 - failed to delete the network as i-2-413-VM was in Error state
- it failed to start. Notice its different vm from the vm on step 1). At
the moment deleteNetwork was called, the vm wasnt' expunged yet. Did you
wait for the vm i-2-413 to be picked up by Expunge thread and moved to
Expunging state? If yes, was it expunged successfully? If no, you have to
find out the references to the vm in the log - search by "to expunge"
pattern in the management server, it will tell you when vm was processed
by the expunge thread.


-Alena.


On 6/13/12 1:51 PM, "Lu Heng" <h.lu@anytimechinese.com> wrote:

>Hi
>
>The story start with I try to setup direct attached network to the
>guest network, in which I create a VM with the shared network I setup,
>but the VM does not start successfully, so I tried to delete the guest
>network I have created, in which I failed because of the vm was not
>destroy, then I try to destroy the VM, but the VM stuck in the
>explunging state:
>
>The network setup was not resolved, I asked in this mailling list what
>should I do if I have public network in the access point with advanced
>network setup, they said I have to create a shared network offering
>with Vlan, in which the public IP has no vlan setup, so I just typed
>tagged in the vlan section,seems not working.
>
>Below is the full story log start with I added a direct attached network:
>
>2012-06-13 08:56:25,401 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Catch Exception: class
>java.lang.NumberFormatException due to
>java.lang.NumberFormatException: For input string: "tagged1"
>java.lang.NumberFormatException: For input string: "tagged1"
>        at
>java.lang.NumberFormatException.forInputString(NumberFormatException.java:
>65)
>2012-06-13 08:56:25,402 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Unable to start i-2-415-VM due to
>java.lang.NumberFormatException: For input string: "tagged1"
>        at
>java.lang.NumberFormatException.forInputString(NumberFormatException.java:
>65)
>2012-06-13 08:56:25,455 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Unable to clean up VBD due to
>You gave an invalid object reference.  The object may have recently
>been deleted.  The class parameter gives the type of reference given,
>and the handle parameter echoes the bad value given.
>        at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(
>CitrixResourceBase.java:930)
>2012-06-13 08:56:25,466 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Unable to clean up VBD due to
>You gave an invalid object reference.  The object may have recently
>been deleted.  The class parameter gives the type of reference given,
>and the handle parameter echoes the bad value given.
>        at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(
>CitrixResourceBase.java:930)
>2012-06-13 08:56:25,478 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Unable to clean up VBD due to
>You gave an invalid object reference.  The object may have recently
>been deleted.  The class parameter gives the type of reference given,
>and the handle parameter echoes the bad value given.
>        at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(
>CitrixResourceBase.java:930)
>2012-06-13 08:56:25,489 WARN  [xen.resource.CitrixResourceBase]
>(DirectAgent-45:null) Unable to cleanup VIF
>You gave an invalid object reference.  The object may have recently
>been deleted.  The class parameter gives the type of reference given,
>and the handle parameter echoes the bad value given.
>        at
>com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(
>CitrixResourceBase.java:938)
>2012-06-13 08:56:25,515 INFO  [xen.resource.XenServer56Resource]
>(DirectAgent-45:null) Catch com.xensource.xenapi.Types$VifInUse:
>failed to destory VLAN eth2 on host
>b9f6241d-cbd6-49a8-916e-dff524569dd4 due to Network has active VIFs
>2012-06-13 08:56:25,517 DEBUG [agent.transport.Request]
>(DirectAgent-45:null) Seq 6-920389063: Processing:  { Ans: , MgmtId:
>229144658033003, via: 6, Ver: v1, Flags: 110,
>[{"StartAnswer":{"vm":{"id":415,"name":"i-2-415-VM","bootloader":"PyGrub",
>"type":"User","cpus":1,"speed":500,"minRam":536870912,"maxRam":536870912,"
>arch":"x86_64","os":"CentOS
>5.3
>(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUs
>e":false,"vncPassword":"7bad48a63e4b14e7","params":{},"disks":[{"id":419,"
>name":"ROOT-415","mountPoint":"/iqn.2006-01.com.openfiler:tsn.2912cb92f82d
>/0","path":"ff3b7226-fd0b-4d0b-8af4-393496d7ee24","size":21474836480,"type
>":"ROOT","storagePoolType":"IscsiLUN","storagePoolUuid":"f7d6960c-84a6-332
>f-9068-27ef0f216b31","deviceId":0},{"id":420,"name":"DATA-415","mountPoint
>":"/iqn.2006-01.com.openfiler:tsn.2912cb92f82d/0","path":"202feab0-b724-4b
>70-b757-32f2b8dfddee","size":5368709120,"type":"DATADISK","storagePoolType
>":"IscsiLUN","storagePoolUuid":"f7d6960c-84a6-332f-9068-27ef0f216b31","dev
>iceId":1},{"id":415,"name":"CentOS
>5.6(64-bit) no GUI
>(XenServer)","size":0,"type":"ISO","storagePoolType":"ISO","deviceId":3}],
>"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"ip":"10.1.1
>.99","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:2e:ae:00:
>04","dns1":"204.13.152.2","broadcastType":"Vlan","type":"Guest","broadcast
>Uri":"vlan://292","isolationUri":"vlan://292","isSecurityGroupEnabled":fal
>se,"name":"Pool-wide
>network associated with
>eth2"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"ip":"46.136
>.134.86","netmask":"255.255.255.0","gateway":"46.136.134.1","mac":"06:4a:b
>4:00:08:3d","dns1":"204.13.152.2","broadcastType":"Vlan","type":"Guest","b
>roadcastUri":"vlan://tagged1","isolationUri":"vlan://tagged1","isSecurityG
>roupEnabled":false,"name":"Pool-wide
>network associated with eth2"}]},"result":false,"details":"Unable to
>start i-2-415-VM due to ","wait":0}}] }
>2012-06-13 08:56:25,517 WARN  [cloud.vm.VirtualMachineManagerImpl]
>(DirectAgent-45:null) Cleanup failed due to Unable to start i-2-415-VM
>due to
>2012-06-13 08:56:25,517 WARN  [cloud.vm.VirtualMachineManagerImpl]
>(Job-Executor-47:job-92) Cleanup failed due to Unable to start
>i-2-415-VM due to
>2012-06-13 08:56:25,590 INFO  [cloud.vm.VirtualMachineManagerImpl]
>(Job-Executor-47:job-92) Unable to start VM on Host[-6-Routing] due to
>Unable to start i-2-415-VM due to
>2012-06-13 08:56:26,159 DEBUG [cloud.capacity.CapacityManagerImpl]
>(Job-Executor-47:job-92) VM state transitted from :Starting to Stopped
>with event: OperationFailedvm's original host id: null new host id:
>null host id before state transition: 6
>2012-06-13 08:56:26,440 DEBUG [cloud.capacity.CapacityManagerImpl]
>(Job-Executor-47:job-92) VM state transitted from :Stopped to Error
>with event: OperationFailedToErrorvm's original host id: null new host
>id: null host id before state transition: null
>2012-06-13 08:56:27,209 INFO  [api.commands.DeployVMCmd]
>(Job-Executor-47:job-92)
>com.cloud.exception.InsufficientServerCapacityException: Unable to
>create a deployment for VM[User|i-2-415-VM]Scope=interface
>com.cloud.dc.DataCenter; id=1
>2012-06-13 08:56:27,210 WARN  [cloud.api.ApiDispatcher]
>(Job-Executor-47:job-92) class com.cloud.api.ServerApiException :
>Unable to create a deployment for VM[User|i-2-415-VM]
>2012-06-13 08:56:27,210 DEBUG [cloud.async.AsyncJobManagerImpl]
>(Job-Executor-47:job-92) Complete async job-92, jobStatus: 2,
>resultCode: 530, result:
>com.cloud.api.response.ExceptionResponse@1660d637
>2012-06-13 09:11:08,903 WARN  [cloud.network.NetworkManagerImpl]
>(Job-Executor-50:job-94) Can't delete the network, not all user vms
>are expunged. Vm VM[User|i-2-413-VM] is in Error state
>2012-06-13 09:11:08,993 WARN  [cloud.api.ApiDispatcher]
>(Job-Executor-50:job-94) class com.cloud.api.ServerApiException :
>Failed to delete network
>2012-06-13 09:11:08,993 DEBUG [cloud.async.AsyncJobManagerImpl]
>(Job-Executor-50:job-94) Complete async job-94, jobStatus: 2,
>resultCode: 530, result:
>com.cloud.api.response.ExceptionResponse@2102e538
>2012-06-13 09:11:09,144 WARN  [cloud.async.AsyncJobManagerImpl]
>(Job-Executor-50:job-94) Unable to unregister active job 94 from JMX
>monitoring
>2012-06-13 11:26:14,160 WARN  [cloud.network.NetworkManagerImpl]
>(Job-Executor-52:job-95) Can't delete the network, not all user vms
>are expunged. Vm VM[User|i-2-413-VM] is in Error state
>2012-06-13 11:26:14,243 WARN  [cloud.api.ApiDispatcher]
>(Job-Executor-52:job-95) class com.cloud.api.ServerApiException :
>Failed to delete network
>
>
>On Wed, Jun 13, 2012 at 10:46 PM, Alena Prokharchyk
><Alena.Prokharchyk@citrix.com> wrote:
>> On 6/13/12 1:38 PM, "Lu Heng" <h.lu@anytimechinese.com> wrote:
>>
>>>Hi
>>>
>>>And btw, I didn't see the maintainace botton for the 3 shared storage,
>>> I only see take snapshoot, snapshoot setting, that's it.
>>>
>>>On Wed, Jun 13, 2012 at 10:35 PM, Lu Heng <h.lu@anytimechinese.com>
>>>wrote:
>>>> Hi
>>>>
>>>> See my reply
>>>>
>>>> On Wed, Jun 13, 2012 at 10:30 PM, Alena Prokharchyk
>>>> <Alena.Prokharchyk@citrix.com> wrote:
>>>>> On 6/13/12 1:24 PM, "Lu Heng" <h.lu@anytimechinese.com> wrote:
>>>>>
>>>>>>Hi
>>>>>>
>>>>>>I find the reason, three vm are in state of "Expunging" for 3 hours
>>>>>>now, no process is being made, anyone has idea what should I do about
>>>>>>these VMs?
>>>>>
>>>>>
>>>>> What is expunge.interval in the setup? If the vms stuck in Expunging
>>>>>state
>>>>> with null Removed field, it indicates some problem happening during
>>>>>the
>>>>> expunge. I suspect it's related with the fact you removed the storage
>>>>>from
>>>>> the xen.
>>>>>
>>>> Vise verse, the situation is, the VM in the expunging state for 3
>>>> hours, and I find no way to delete them, so I try to delete the
>>>> storage.
>>
>>
>> So you deleted the VHDs per each volume, not the storage itself, please
>> confirm. And we should never do it as there is no volume sync, so if the
>> volume is removed on the backend, cloudStack has no idea it has
>>happened.
>>
>> The original issue is - vm got stuck in Expunging state. We have to
>>figure
>> out this one first. Please send the logs starting from the time the
>> Destroy command was executed for the Vm. And mention the vm name.
>>
>>
>> -Alena.
>>
>>
>>
>>
>>
>>>>
>>>> What should I do to fix this now?
>>>>>
>>>>>
>>>>>>
>>>>>>Another question is, I didn't see the delete botton with shared
>>>>>>storage, so I delete it from the xencenter, but after that I still
>>>>>>see
>>>>>>the deleted storage exists in the UI, what suppose to be done?
>>>>>
>>>>>
>>>>> Delete storage pool consists of 2 steps:
>>>>>
>>>>> * enable maintenance for the shared storage
>>>>> * after the maintenance is completed, the Delete button will be
>>>>>presented
>>>>> to the admin; and the storage can be removed.
>>>>>
>>>>> You are not supposed to delete primary storage on the backend;
>>>>>cloudStack
>>>>> has no idea if you do things like that, and it expects primary
>>>>>storage
>>>>>to
>>>>> be functioning.
>>>>>
>>>>> So when vm expunge happens, we see that the storage is in UP state,
>>>>>and
>>>>> try to send the destroy volume command to the Xen host. But the
>>>>>command
>>>>> fails there because there is no such primary storage exist.
>>>>>
>>>>> Let me know if its your case.
>>>>>
>>>>> -Alena.
>>>>>
>>>>>
>>>>>>
>>>>>>On Wed, Jun 13, 2012 at 10:08 PM, Alena Prokharchyk
>>>>>><Alena.Prokharchyk@citrix.com> wrote:
>>>>>>> On 6/13/12 12:55 PM, "Lu Heng" <h.lu@anytimechinese.com>
wrote:
>>>>>>>
>>>>>>>>Hi
>>>>>>>>
>>>>>>>>Update:
>>>>>>>>
>>>>>>>>The zone has been stoped. all VM has been destoryed , the
route vm
>>>>>>>>has
>>>>>>>>been stoped(no option for destory).
>>>>>>>>
>>>>>>>>Still get error fail to delete network.
>>>>>>>
>>>>>>>
>>>>>>> Can you send management server log file for debugging.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Alena.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>On Wed, Jun 13, 2012 at 7:23 PM, Alena Prokharchyk
>>>>>>>><Alena.Prokharchyk@citrix.com> wrote:
>>>>>>>>> On 6/13/12 10:18 AM, "Tamas Monos" <tamasm@veber.co.uk>
wrote:
>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>Also I think the problem is when you are trying to
delete the
>>>>>>>>>>network
>>>>>>>>>>the
>>>>>>>>>>router is powered off.
>>>>>>>>>>Therefore when you click delete it wants to remove
the firewall
>>>>>>>>>>rules
>>>>>>>>>>for
>>>>>>>>>>that network but the router is not running so the
command fails
>>>>>>>>>>which
>>>>>>>>>>fails the delet operation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also not a problem. The rules can be removed when the
router is
>>>>>>>>>stopped.
>>>>>>>>> In this case the rules are removed in DB only. When the
router
>>>>>>>>>starts
>>>>>>>>>up,
>>>>>>>>> we start clean (cleanup all the rules) and re-apply all
the rules
>>>>>>>>>from
>>>>>>>>>the
>>>>>>>>> DB. So the removed rules won't exist  on the router.
>>>>>>>>>
>>>>>>>>> -Alena.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Regards
>>>>>>>>>>
>>>>>>>>>>Tamas Monos                                      
        DDI
>>>>>>>>>>+44(0)2034687012
>>>>>>>>>>Chief Technical
>>>>>>>>>>Office
>>>>>>>>>>+44(0)2034687000
>>>>>>>>>>Veber: The Hosting Specialists               Fax
>>>>>>>>>>+44(0)871
>>>>>>>>>>522
>>>>>>>>>>7057
>>>>>>>>>>http://www.veber.co.uk
>>>>>>>>>>
>>>>>>>>>>Follow us on Twitter: www.twitter.com/veberhost
>>>>>>>>>>Follow us on Facebook: www.facebook.com/veberhost
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>-----Original Message-----
>>>>>>>>>>From: Tamas Monos [mailto:tamasm@veber.co.uk]
>>>>>>>>>>Sent: 13 June 2012 18:15
>>>>>>>>>>To: cloudstack-dev@incubator.apache.org
>>>>>>>>>>Subject: RE: Can not delete network
>>>>>>>>>>
>>>>>>>>>>Hi,
>>>>>>>>>>
>>>>>>>>>>I don't think you can delete a network if a router-vm
for that
>>>>>>>>>>network
>>>>>>>>>>exists.
>>>>>>>>>>If you are trying to delete a network that network
should not
>>>>>>>>>>have
>>>>>>>>>>any
>>>>>>>>>>VMs.
>>>>>>>>>>Delete the associated router from the management interface
then
>>>>>>>>>>delete
>>>>>>>>>>the network.
>>>>>>>>>>This method works for me on 3.0.2
>>>>>>>>>>
>>>>>>>>>>Regards
>>>>>>>>>>
>>>>>>>>>>Tamas Monos                                      
        DDI
>>>>>>>>>>+44(0)2034687012
>>>>>>>>>>Chief Technical
>>>>>>>>>>Office
>>>>>>>>>>+44(0)2034687000
>>>>>>>>>>Veber: The Hosting Specialists               Fax
>>>>>>>>>>+44(0)871
>>>>>>>>>>522
>>>>>>>>>>7057
>>>>>>>>>>http://www.veber.co.uk
>>>>>>>>>>
>>>>>>>>>>Follow us on Twitter: www.twitter.com/veberhost Follow
us on
>>>>>>>>>>Facebook:
>>>>>>>>>>www.facebook.com/veberhost
>>>>>>>>>>
>>>>>>>>>>-----Original Message-----
>>>>>>>>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>>>>>>>>Sent: 13 June 2012 18:12
>>>>>>>>>>To: cloudstack-dev@incubator.apache.org
>>>>>>>>>>Subject: Re: Can not delete network
>>>>>>>>>>
>>>>>>>>>>Can you please check the management server log to
see if there is
>>>>>>>>>>more
>>>>>>>>>>detailed exception in there.
>>>>>>>>>>
>>>>>>>>>>-Alena.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>On 6/13/12 9:50 AM, "Lu Heng" <h.lu@anytimechinese.com>
wrote:
>>>>>>>>>>
>>>>>>>>>>>Hi
>>>>>>>>>>>
>>>>>>>>>>>I have problem deleting guest network, everytime
I tried it
>>>>>>>>>>>always
>>>>>>>>>>>shows "fail to delete network)
>>>>>>>>>>>
>>>>>>>>>>>2012-06-13 11:26:14,243 WARN  [cloud.api.ApiDispatcher]
>>>>>>>>>>>(Job-Executor-52:job-95) class com.cloud.api.ServerApiException
>>>>>>>>>>>:
>>>>>>>>>>>Failed to delete network
>>>>>>>>>>>
>>>>>>>>>>>Any idea how it happens?
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>Kind regards.
>>>>>>>>>>>Lu
>>>>>>>>>>>
>>>>>>>>>>>This transmission is intended solely for the addressee(s)
shown
>>>>>>>>>>>above.
>>>>>>>>>>>It may contain information that is privileged,
confidential or
>>>>>>>>>>>otherwise protected from disclosure. Any review,
dissemination
>>>>>>>>>>>or
>>>>>>>>>>>use
>>>>>>>>>>>of this transmission or its contents by persons
other than the
>>>>>>>>>>>intended
>>>>>>>>>>>addressee(s) is strictly prohibited. If you have
received this
>>>>>>>>>>>transmission in error, please notify this office
immediately and
>>>>>>>>>>>e-mail
>>>>>>>>>>>the original at the sender's address above by
replying to this
>>>>>>>>>>>message
>>>>>>>>>>>and including the text of the transmission received.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>--
>>>>>>>>--
>>>>>>>>Kind regards.
>>>>>>>>Lu
>>>>>>>>
>>>>>>>>This transmission is intended solely for the addressee(s)
shown
>>>>>>>>above.
>>>>>>>>It may contain information that is privileged, confidential
or
>>>>>>>>otherwise protected from disclosure. Any review, dissemination
or
>>>>>>>>use
>>>>>>>>of this transmission or its contents by persons other than
the
>>>>>>>>intended addressee(s) is strictly prohibited. If you have
received
>>>>>>>>this transmission in error, please notify this office immediately
>>>>>>>>and
>>>>>>>>e-mail the original at the sender's address above by replying
to
>>>>>>>>this
>>>>>>>>message and including the text of the transmission received.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>--
>>>>>>--
>>>>>>Kind regards.
>>>>>>Lu
>>>>>>
>>>>>>This transmission is intended solely for the addressee(s) shown
>>>>>>above.
>>>>>>It may contain information that is privileged, confidential or
>>>>>>otherwise protected from disclosure. Any review, dissemination or
use
>>>>>>of this transmission or its contents by persons other than the
>>>>>>intended addressee(s) is strictly prohibited. If you have received
>>>>>>this transmission in error, please notify this office immediately
and
>>>>>>e-mail the original at the sender's address above by replying to this
>>>>>>message and including the text of the transmission received.
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Kind regards.
>>>> Lu
>>>>
>>>> This transmission is intended solely for the addressee(s) shown above.
>>>> It may contain information that is privileged, confidential or
>>>> otherwise protected from disclosure. Any review, dissemination or use
>>>> of this transmission or its contents by persons other than the
>>>> intended addressee(s) is strictly prohibited. If you have received
>>>> this transmission in error, please notify this office immediately and
>>>> e-mail the original at the sender's address above by replying to this
>>>> message and including the text of the transmission received.
>>>
>>>
>>>
>>>--
>>>--
>>>Kind regards.
>>>Lu
>>>
>>>This transmission is intended solely for the addressee(s) shown above.
>>>It may contain information that is privileged, confidential or
>>>otherwise protected from disclosure. Any review, dissemination or use
>>>of this transmission or its contents by persons other than the
>>>intended addressee(s) is strictly prohibited. If you have received
>>>this transmission in error, please notify this office immediately and
>>>e-mail the original at the sender's address above by replying to this
>>>message and including the text of the transmission received.
>>>
>>
>>
>
>
>
>--
>--
>Kind regards.
>Lu
>
>This transmission is intended solely for the addressee(s) shown above.
>It may contain information that is privileged, confidential or
>otherwise protected from disclosure. Any review, dissemination or use
>of this transmission or its contents by persons other than the
>intended addressee(s) is strictly prohibited. If you have received
>this transmission in error, please notify this office immediately and
>e-mail the original at the sender's address above by replying to this
>message and including the text of the transmission received.
>



Mime
View raw message