cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Monos <>
Subject RE: Get system VM out of hung migrating state
Date Thu, 17 Jan 2013 11:13:51 GMT

I am not sure if I will be able to help you at all, but will try.
Are you using 3.0.2? Can you just not kill the Console Proxy VM from CS interface with 'Destroy'
and 3.0.2 will automatically re-create it.
If you are using 3.0.2+ do not attempt this as later versions have a tendency not to rebuilt
systemVM (4.0.x)
If it does not allow you because of the VM state you could go and edit the database and mark
the systemVM as Destroyed and put a date next to it. Then restart the management server.
Off course MAKE a BACKUP before you'd do that.

I had to do a similar thing when my test environment self-destructed :/

I'm running 3.0.2 with vmware for nearly a year now non-stop and my systemVMs are fine and
CS too.
I was always struggling with Xen/KVM cluster because CS wants to manage the cluster and does
not really do a great job in that respect, so I went vmware way where CS has the 'hands-off'
approach and let's vcenter to do the clustering job.
I know this does not help you right now, just a thing to consider.


Tamas Monos                                               DDI         +44(0)2034687012
Chief Technical                                             Office    +44(0)2034687000
Veber: The Hosting Specialists               Fax         +44(0)871 522 7057

Follow us on Twitter:
Follow us on Facebook:

-----Original Message-----
From: Nik Martin [] 
Sent: 17 January 2013 05:09
Subject: Get system VM out of hung migrating state

Ok, the saga of the hung console proxy VM continues, and I really need to get this cleaned
up.  After a hard reboot and having to force the hypervisor online via the database, my consol
proxy VM is still hung in migrating state, so CS essentially ignores it:

2013-01-16 23:03:38,763 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-492:null) Seq 3-1375993858: Executing request
2013-01-16 23:03:39,155 WARN  [xen.resource.CitrixResourceBase]
(DirectAgent-492:null) The VM is now missing marking it as Stopped v-11-VM
2013-01-16 23:03:39,155 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-492:null) Seq 3-1375993858: Response Received:
2013-01-16 23:03:39,155 DEBUG [agent.transport.Request]
(DirectAgent-492:null) Seq 3-1375993858: Processing:  { Ans: , MgmtId: 
130577622632, via: 3, Ver: v1, Flags: 10, [{"ClusterSyncAnswer":{"_clusterId":1,"_newStates":{"v-11-VM":{"t":"4bafdf37-d0e7-4652-90dd-afa6f9099598","u":"Stopped"}},"_isExecuted":false,"result":true,"wait":0}}]
2013-01-16 23:03:39,160 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-492:null) VM v-11-VM: cs state = Migrating and realState = Stopped
2013-01-16 23:03:39,160 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-492:null) VM v-11-VM: cs state = Migrating and realState = Stopped
2013-01-16 23:03:39,160 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-492:null) Skipping vm in migrating state: 
2013-01-16 23:03:44,793 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Ping from 13

How do I get this VM back online, and do any developers need anything to try and dtermine
what is wrong?  The status of system VMs and hosts is VERY fragile in CS, and there seems
to be little error recovery in place.  CS shoudl not just ignore a VM in a migrating state.
 I have a migratewait setting of 3600 seconds, but this VM has been in this state  > 24
hours, after multiple management restarts.



Nik Martin
nfina Technologies, Inc.
+ x1003
Relentless Reliability

View raw message