cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Marshall <jms....@hotmail.co.uk>
Subject Re: Failover for VMs
Date Thu, 29 Mar 2018 08:40:58 GMT
Hi Paul


I did make some progress with this and seem to remember that after it said Recovered it then
went back to Suspect and finally Fenced.


I am going to rerun a lot of the tests after changing some of the kvm_ha_ timers to try and
speed things up a bit.


Will update here after I have run tests to check if that is what I should be seeing.


Many thanks


Jon


________________________________
From: Paul Angus <paul.angus@shapeblue.com>
Sent: 28 March 2018 20:01
To: users@cloudstack.apache.org
Subject: Re: Failover for VMs

Ah.

Did you wait after the node said recovered?

That message is spurious. I've seen it also. It should say recovering.   at that time.

________________________________
From: Jon Marshall <jms.123@hotmail.co.uk>
Sent: Tuesday, 27 March 2018 10:42 am
To: users@cloudstack.apache.org
Subject: Re: Failover for VMs

Just as an update to this before I forget what I did :) -


I used "echo c /proc/sysrq-trigger" on one of the compute nodes and there was no VM failover.
 Instead HA reported suspect and then IPMI rebooted the machine, it came back online and the
VM started responding to pings again.  IPMI is out of band so that seems to be reasonable
behaviour but no use in testing HA.


Next I just pulled all 3 NIC cables  from the same compute node and again HA reported suspect.
 Again IPMI rebooted but then HA state changed to "Recovered" which I don't understand as
the NIC cables were still disconnected so VM was not reachable and no failover.


I don't understand how it can think the node is recovered as apart from the IPMI out of band
connection there are no network connections to this server.


Finally pulled power lead and this time HA went from suspect to Fencing and then stayed that
way. Again no VM failover.   This makes sense as no power means IPMI cannot reboot server
so it never moves to Fenced I assume. Again no failover.


I am wondering if it is to do with out of band IPMI or the way I have the NICs setup.  The
management node only has one NIC in the management network but I assume this is okay.


I may try reloading with CS v4.9 and just try failover without the new HA KVM to see if I
see anything different.



Jon


________________________________
From: Jon Marshall <jms.123@hotmail.co.uk>
Sent: 27 March 2018 10:10
To: users@cloudstack.apache.org
Subject: Re: Failover for VMs


Thanks Paul, will pick up after Easter break.

Doing some more testing with HA KVM at the moment so any progress will update this thread


i
________________________________
From: Paul Angus <paul.angus@shapeblue.com
Sent: 27 March 2018 10:07
To: users@cloudstack.apache.org
Subject: RE: Failover for VMr
Jon,

I've been updating the Ansible to move our physical hosts from Centos6 to Centos7, now that's
done I'll run through an HA setup and post answers (probably after easter break).

paul.angus@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




paul.angus@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-----Original Message-----
From: Jon Marshall <jms.123@hotmail.co.uk>
Sent: 27 March 2018 09:19
To: users@cloudstack.apache.org
Subject: Failover for VMs

After 3 weeks of trying multiple different setups I still have not managed to get a VM to
failover between compute nodes and am just running out of ideas.


I have 3 compute nodes each with 3 NICS (management, VMs traffic, storage), one management
node with just a single NIC connection in the management network and a separate NFS server.


I have tried with and without the new Host HA KVM in CS v4.11 as from what I have read even
without enabling the new Host HA KVM when you power off or reboot a compute node your VMs
should still migrate.


I have tried powering off a compute node, pulling the power lead, removing the management
and NFS network cables and the management server just seems to carry on as if nothing has
happened.


Could someone explain exactly how HA is meant to work so I can look at where it is going wrong.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message