cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaime Orlando Rojas Sanchez <jaime.ro...@kumo.com.co>
Subject RE: VM stuck in a failing Host *RESOLVED
Date Fri, 21 Aug 2015 15:24:13 GMT
Hello, finally we fixed the problem:

1.	From CS state VM was change to "Stopped" from DB, last_host_ID was marked as NULL
2.	Then on Xenservers we did
a.	Auto power on for VM :
xe vm-param-set uuid=[UUID] other-config:auto_poweron=true
b.	Pool DB sync.
xe pool-sync-database
c.	Forced Power off
xe vm-reset-powerstate uuid=[UUID] -force
d.	Check VM state
xe vm-list name-label=i-574-14584-VM (state: healt)
3.	From CS it was turned on and it started in a different host.

Not sure exactly which command made the difference on XS but it was clear that once the VM
was stuck it turned a XS issue further than CS handled. 

Regards / Cordialmente,

Jaime O. Rojas S.
Technology Manager
jaime.rojas@kumo.com.co
Mobile: +57 301-3382382
Office: +57-1-8766767 x215

-----Mensaje original-----
De: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co] 
Enviado el: jueves, 20 de agosto de 2015 7:07 p. m.
Para: <users@cloudstack.apache.org>
Asunto: Re: VM stuck in a failing Host

Hello, I already tried he force stop check, unfortunately it didn't work VM keeps starting
in failing host. I am trying to force vm to stop from xen server with no results.

[root@dc1fdtptgcx02 /]# xe vm-shutdown name-label=i-574-14584-VM --force You attempted an
operation which involves a host which could not be contacted.
host: 53109eef-2f53-4f0e-a763-68817d573bd9 (DC1FDTPTGCX03

Cordialmente,

Jaime Rojas

Sent from my iPhone

On 20/08/2015, at 4:03 p.m., Somesh Naidu <Somesh.Naidu@citrix.com<mailto:Somesh.Naidu@citrix.com>>
wrote:

Thanks I just realized that the job was a stop command and not start command that I was looking
for.

The stop command failed to stop the VM leaving the VM in Running state:
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-21:null) submit
async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 14584,
cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","id":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","sessionkey":"hwnxmM0He9EXs2craugKg3XyWL4\u003d","cmdEventType":"VM.STOP","ctxUserId":"2","httpmethod":"GET","_":"1440067422009","ctxAccountId":"2","ctxStartEventId":"38650949","forced":"true"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 139549854171544, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2015-08-20 05:45:27,605 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-30:job-45973
= [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state transitted from :Running to Stopping
with event: StopRequestedvm's original host id: 599 new host id: 599 host id before state
transition: 599
2015-08-20 05:45:27,617 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-30:job-45973
= [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException:
Resource [Host:599] is unreachable: Host 599: Host with specified id is not in the right state:
Disconnected
2015-08-20 05:45:32,654 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-30:job-45973
= [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) VM state transitted from :Stopping to Running
with event: OperationFailedvm's original host id: 599 new host id: 599 host id before state
transition: 599
2015-08-20 05:45:33,683 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-45973
= [ d3ed77b7-534e-4a3b-9038-a66359162087 ]) Complete async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087
], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to stop vm

I can't tell why the VM keeps starting on the failed host unless I have a look at the logs
from a start command.

But I think there is a good chance that if you check the "Force Stop" check box when stopping
that VM and then try to start it again.

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 4:40 PM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Subject: RE: VM stuck in a failing Host

Hello,

I am attaching the logs

http://jrojas-test.s3.kumo.com.co/management-server.log?AWSAccessKeyId=0099c1623d6704f9f5d5&Expires=1440711480&Signature=CFYwowzsJ3x%2B7XXfpAHDLM%2F022A%3D&x-amz-pt=N2ZiYjcyOTk4MTQ0MDEwMzEwNTQ5MA

We have a second manage server but it doesn't execute any task.

Regards / Cordialmente,

Jaime O. Rojas S.
Technology Manager
jaime.rojas@kumo.com.co<mailto:jaime.rojas@kumo.com.co>
Mobile: +57 301-3382382
Office: +57-1-8766767 x215

-----Mensaje original-----
De: Somesh Naidu [mailto:Somesh.Naidu@citrix.com] Enviado el: jueves, 20 de agosto de 2015
2:17 p. m.
Para: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Asunto: RE: VM stuck in a failing Host

Jaime,

Those are not complete logs I need. I need complete logs (no grep) between "2015-08-20 05:45:27,513"
and when the job (job-45973) finished (I can't tell from the logs you shared earlier).

Also, do you have more than 1 management server?

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 2:41 PM
To: <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
Subject: Re: VM stuck in a failing Host

We ran the command without the space with no results.

In the second message I sent, I pasted the logs. You can find it in this mail chain at the
beginning.

Cordialmente,

Jaime Rojas

Sent from my iPhone

On 20/08/2015, at 1:37 p.m., Somesh Naidu <Somesh.Naidu@citrix.com<mailto:Somesh.Naidu@citrix.com><mailto:Somesh.Naidu@citrix.com>>
wrote:

We do not need to take a snapshot. Just extract the VHD from storage (you could use vdi-copy
if it is on block storage) and then register it in CS as a template from UI.

xe vm-list name-label= i-574-14584-VM

The command you tried above has space after "=". Did you try without the space?

Also, can you share the management server logs that capture the VM start job so we can take
a look?

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 1:59 PM
To: <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>>
Subject: Re: VM stuck in a failing Host

We already did all of that and VM keeps trying to start on failed host. Template is not possible
we tried to take a snapshot but it return an error.

Cordialmente,

Jaime Rojas



On 20/08/2015, at 12:19 p.m., Somesh Naidu <Somesh.Naidu@citrix.com<mailto:Somesh.Naidu@citrix.com><mailto:Somesh.Naidu@citrix.com>>
wrote:

I do not believe that should happen as the VM is primarily tied to the storage. But just to
avoid taking any chances, we could do this:
1. Put the failed host in to maintenance.
2. Set host_id and last_host_id to NULL in vm_instance table for that VM.
3. Make sure the XS pool doesn't have any stale data for this VM. Look for the VM and associated
VDI in the pool one more time. I noticed there is a space "name-label= i-574-14584-VM" after
the equal operator, remove that space and try again or use XenCenter.

If #3 doesn't raise any flags, go ahead and try starting the VM again.

There is also another alternative, you could register the root disk of the VM as a template
and launch a new VM from that template.

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 1:00 PM
To: <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>>
Subject: Re: VM stuck in a failing Host

Yes we could , but I'm not sure if maybe I lost the VM because it is tied to that host.

Cordialmente,

Jaime Rojas



On 20/08/2015, at 11:47 a.m., Somesh Naidu <Somesh.Naidu@citrix.com<mailto:Somesh.Naidu@citrix.com><mailto:Somesh.Naidu@citrix.com>>
wrote:

Can you not remove the failed host from CCP and XS cluster?

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 12:38 PM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>
Subject: RE: VM stuck in a failing Host

Hello,

state/status of the failed host in CS? = Disconnected

We run the command in the 2 remaining host with no results.

[root@dc1fdtptgcx04 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx04 /]#

[root@dc1fdtptgcx02 /]# xe vm-list name-label= i-574-14584-VM
[root@dc1fdtptgcx02 /]#



Regards / Cordialmente,

Jaime O. Rojas S.
Technology Manager
jaime.rojas@kumo.com.co<mailto:jaime.rojas@kumo.com.co><mailto:jaime.rojas@kumo.com.co>
Mobile: +57 301-3382382
Office: +57-1-8766767 x215

-----Mensaje original-----
De: Somesh Naidu [mailto:Somesh.Naidu@citrix.com] Enviado el: jueves, 20 de agosto de 2015
11:00 a. m.
Para: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>
Asunto: RE: VM stuck in a failing Host

Quick question, what is the state/status of the failed host in CS? Also, look up the particular
VM on XS pool (xe vm-list name-label= i-574-14584-VM), what does it say?

Regards,
Somesh


-----Original Message-----
From: Jaime Orlando Rojas Sanchez [mailto:jaime.rojas@kumo.com.co]
Sent: Thursday, August 20, 2015 11:37 AM
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>
Subject: RE: VM stuck in a failing Host

Following the logs when I click 'run' in ACS after did the following in the DB


-          Change the state to 'stopped'

-          Change host ID to a working host

-          Change last host ID to a working host

-          Check VR is up and running on a working host


-bash-4.1# tail -f management-server.log | grep 14584
2015-08-20 05:45:27,513 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-21:null) submit
async job-45973 = [ d3ed77b7-534e-4a3b-9038-a66359162087 ], details: AsyncJobVO {id:45973,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 14584,
cmd: org.apache.cloudstack.api.command.user.vm.StopVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","id":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","sessionkey":"hwnxmM0He9EXs2craugKg3XyWL4\u003d","cmdEventType":"VM.STOP","ctxUserId":"2","httpmethod":"GET","_":"1440067422009","ctxAccountId":"2","ctxStartEventId":"38650949","forced":"true"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 139549854171544, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2015-08-20 05:46:33,404 DEBUG [agent.transport.Request] (Job-Executor-31:job-45974 = [ 9996fc72-fb83-4e5d-94c5-886396dac536
]) Seq 792-1073807425: Sending  { Cmd , MgmtId: 139549854171544, via: 792, Ver: v1, Flags:
100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/574/57764/28586b35-cb45-4565-bd9b-7aa46a2898da","volume":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.4.65/vol/secondary_clpr","_role":"Image"}},"vmName":"i-574-14584-VM","name":"srvrasautos2_ROOT-14584_20141007233517","hypervisorType":"XenServer","id":11831}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/574/599","uuid":"70d21214-33d0-49e0-8b45-c7702b0fe579","id":599,"format":"RAW","accountId":574,"hvm":true,"displayText":"templateras","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.4.65/vol/secondary_clpr","_role":"Image"}},"name":"248e2097b-4af7-38f7-a851-029ef11f52cc","hypervisorType":"XenServer"}},"executeInSequence":true,"wait":10800}}]
}
2015-08-20 05:49:03,212 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-1:null) submit
async job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ], details: AsyncJobVO {id:45975,
userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 14584,
cmd: org.apache.cloudstack.api.command.user.vm.StartVMCmd, cmdOriginator: null, cmdInfo: {"response":"json","id":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","sessionkey":"hwnxmM0He9EXs2craugKg3XyWL4\u003d","cmdEventType":"VM.START","ctxUserId":"2","httpmethod":"GET","_":"1440067641673","ctxAccountId":"2","ctxStartEventId":"38651246"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 139549854171544, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2015-08-20 05:49:05,821 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-32:job-45975
= [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking VirtualRouter to prepare for Nic[194602-14584-5d82a92d-b828-45bc-882a-b5ce17401812-172.16.100.244]
2015-08-20 05:49:08,947 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-32:job-45975
= [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) Asking VirtualRouter to prepare for Nic[194621-14584-null-172.16.180.35]
2015-08-20 05:49:10,181 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-32:job-45975
= [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No need to recreate the volume: Vol[13270|vm=14584|DATADISK],
since it already has a pool assigned: 208, adding disk to VM
2015-08-20 05:49:10,184 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-32:job-45975
= [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f ]) No need to recreate the volume: Vol[57764|vm=14584|ROOT],
since it already has a pool assigned: 211, adding disk to VM
2015-08-20 05:49:10,271 DEBUG [agent.transport.Request] (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f
]) Seq 595-838074847: Sending  { Cmd , MgmtId: 139549854171544, via: 595, Ver: v1, Flags:
100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS
6.0 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"hostIp":"172.16.1.11","executeInSequence":true,"wait":0}}]
}
2015-08-20 05:49:10,273 DEBUG [agent.transport.Request] (Job-Executor-32:job-45975 = [ 8ac23585-989d-4e3d-bcb9-3d3602842b8f
]) Seq 595-838074847: Executing:  { Cmd , MgmtId: 139549854171544, via: 595, Ver: v1, Flags:
100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS
6.0 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"hostIp":"172.16.1.11","executeInSequence":true,"wait":0}}]
}
2015-08-20 05:49:10,402 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-434:null) VM
i-574-14584-VM is runing on host 53109eef-2f53-4f0e-a763-68817d573bd9
2015-08-20 05:49:10,403 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-434:null) The
VM is in stopped state, detected problem during startup : i-574-14584-VM
2015-08-20 05:49:10,404 DEBUG [agent.transport.Request] (DirectAgent-434:null) Seq 595-838074847:
Processing:  { Ans: , MgmtId: 139549854171544, via: 595, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":14584,"name":"i-574-14584-VM","bootloader":"PyGrub","type":"User","cpus":2,"minSpeed":525,"maxSpeed":2100,"minRam":4294967296,"maxRam":4294967296,"arch":"x86_64","os":"CentOS
6.0 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"63729fa8c6c9ecae","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:true;apic:true;pae:true;nx:false","Message.ReservedCapacityFreed.Flag":"true","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"98227dc9-682e-4f42-87e1-bd4b8045c7c9","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"71095933-5ce2-4786-8527-1dbe11876004","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_2000GB_ADV_SAS","id":208,"poolType":"PreSetup","host":"localhost","path":"/3PAR_2000GB_ADV_SAS","port":0}},"name":"Datos","size":214748364800,"path":"69451be5-bd65-41d4-b465-08933d393498","volumeId":13270,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":13270,"hypervisorType":"XenServer"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a15d0923-0a25-408f-9d10-fd5d47b3fef9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3PAR_3000GB_ADV_SATA1","id":211,"poolType":"PreSetup","host":"localhost","path":"/3PAR_3000GB_ADV_SATA1","port":0}},"name":"ROOT-14584","size":107374182400,"path":"c7a8eebc-7750-455c-804f-64c0d66cb4f4","volumeId":57764,"vmName":"i-574-14584-VM","accountId":574,"format":"VHD","id":57764,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"040aaa12-4338-4181-bb60-9dc07aa804e8","ip":"172.16.100.244","netmask":"255.255.252.0","gateway":"172.16.100.1","mac":"02:00:3b:b1:00:16","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://3042","isolationUri":"vlan://3042","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"3ef8c0d3-83c0-4569-990a-577ecd21f707","ip":"172.16.180.35","netmask":"255.255.255.240","gateway":"172.16.180.33","mac":"06:07:8c:00:0f:0d","dns1":"66.165.160.179","dns2":"66.165.160.180","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://182","isolationUri":"vlan://182","isSecurityGroupEnabled":false,"name":"VLAN3000-3010"}]},"host_guid":"53109eef-2f53-4f0e-a763-68817d573bd9","result":true,"details":"VM
i-574-14584-VM is runing on host 53109eef-2f53-4f0e-a763-68817d573bd9","wait":0}}] }
2015-08-20 05:49:19,614 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-41:null) Detecting
a new state but couldn't find a old state so adding it to the changes: i-574-14584-VM
2015-08-20 05:49:19,615 DEBUG [agent.transport.Request] (DirectAgent-41:null) Seq 566-1609629709:
Processing:  { Ans: , MgmtId: 139549854171544, via: 566, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":5,"_newStates":{"i-574-14584-VM":{"t":"53109eef-2f53-4f0e-a763-68817d573bd9","u":"Running","v":"viridian:true;acpi:true;apic:true;pae:true;nx:false"}},"_isExecuted":false,"result":true,"wait":0}}]
}
2015-08-20 05:49:19,627 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-41:null) VM
i-574-14584-VM: cs state = Running and realState = Running
2015-08-20 05:49:19,627 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-41:null) VM
i-574-14584-VM: cs state = Running and realState = Running

Regards / Cordialmente,

Jaime O. Rojas S.
Technology Manager
jaime.rojas@kumo.com.co<mailto:jaime.rojas@kumo.com.co><mailto:jaime.rojas@kumo.com.co><mailto:jaime.rojas@kumo.com.co>
Mobile: +57 301-3382382
Office: +57-1-8766767 x215

De: Jaime Orlando Rojas Sanchez
Enviado el: jueves, 20 de agosto de 2015 9:54 a. m.
Para: 'users@cloudstack.apache.org<mailto:users@cloudstack.apache.org><mailto:users@cloudstack.apache.org>'
Asunto: VM stuck in a failing Host

Hello,

We have a 4.2.1 ACS, running on XenServer 6.2.0, we have a zone with a pool of 3 host, yesterday
1 host crash and OS get corrupted. I think we lost that host and have to reinstall it, but
the issue is that we had a couple of VM and VR running on that host. The failing host  was
the master of the pool, so once it fails all the pool was disconnected, we change the master
role and recover pool management from Xencenter and ACS, once we did it a VM moved to the
remaining host, all VR and 1 VM kept stuck in failing host.

In DB we see the VR and VM running, even if the host was marked as down and maintenance. We
changed the VR state to 'stopped' and change de "last host ID" and "Host ID" to a working
host. Once we did it we were able to destroy the VR and recreate them with successful results,
they came up on working host. If we change only the state, the VR couldn't be destroyed. Here
we workaround with the 70% of the outage, BUT one VM remain stuck to the host, we change the
state, the last host ID, but once we press start, it "runs" on the failing host and the VM
appears as running even if it doesn't. Any suggestion to force the VM to start in a different
host and remove it from the failing host? This is a critical VM, we hope somebody else could
give us a hand.

Regards / Cordialmente,

Jaime O. Rojas S.
Technology Manager
jaime.rojas@kumo.com.co<mailto:jaime.rojas@kumo.com.co><mailto:jaime.rojas@kumo.com.co><mailto:jaime.rojas@kumo.com.co>
Mobile: +57 301-3382382
Office: +57-1-8766767 x215


Email asegurado por Check Point

Email asegurado por Check Point

Email asegurado por Check Point

Email asegurado por Check Point

Email asegurado por Check Point

Email asegurado por Check Point



Email asegurado por Check Point


Mime
View raw message