cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Marshall <jms....@hotmail.co.uk>
Subject Re: KVM HostHA
Date Tue, 13 Mar 2018 11:23:38 GMT
Update on below.


I pulled the NICs for both management and storage from cnode 1.


1) The UI immediately showed the power state as Unknown but the state was Up.

2) The HA state on cnode1 showed as suspect. The HA state on cnode2 showed as available.

3) After about 4 mins the state on cnode1 went from Up to Alert

4) The HA state on cnode1 showed as Fencing and the HA state on cnode2 showed as Ineligible.


The HA enabled VMs on cnode1 never switched over to the working node cnode2.


Any ideas ?


________________________________
From: Jon Marshall <jms.123@hotmail.co.uk>
Sent: 13 March 2018 10:50
To: users@cloudstack.apache.org
Subject: Re: KVM HostHAtot stop the server responding to an ipmitool request on the manager
eg -


"ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"


from the management server got an answer saying the chassis power was on so CS never registered
the compute node as down.


I am obviously doing something wrong but cannot work it out.


The management server has one NIC - 172.16.7.4


Each compute node has 3 NICs -


                                       cnode1                        cnode2


mangement NIC        172.16.7.5                   172.16.7.6

vm NIC                      172.16.6.130                 172.16.6.131

storage -                     172.16.250.4               172.16.250.5


Dell LOM (for Idrac)   172.16.7.29                172.16.7.30


the dell LOM IPs are the ones used to configure OOBM  in the UI



If I pull the storage NIC presumably nothing will happen as the ipmitool check is running
across the management NIC so I need to pull both ?

My understanding of host HA was the management server monitored the compute nodes using ipmitool
and if it did not get a response because the host was down it would fence off that host and
move the VMs to an active compute node.

This is obviously too simplistic so could someone explain how it is meant to work and what
it is protecting against ?

________________________________
From: Paul Angus <paul.angus@shapeblue.com>
Sent: 13 March 2018 07:01
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA

Hi all,

One small note, unplugging the management NIC will only cause an HA event if the storage is
running over that NIC also.

Is the storage is over a separate NIC then, the guest VMs will continue to run when the mgmt.
NIC is unplugged, Host HA will detect the disk activity and conclude that there is nothing
it can do, as the VMs are still running other than mark the hosts as degraded.


Kind regards,

Paul Angus

paul.angus@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-----Original Message-----
From: Parth Patel <parthpatel2597@gmail.com>
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA

>
> Hi Jon,
>
> As I said, in my case, making the host HA didn't work but by just
> having a HA VM running on host and executing - (WARNING) "echo c >
> /proc/sysrq-trigger" to simulate a kernel crash on host, the
> management server registered it as down and started the VM on another
> host. I know I've suggested this before but I insist you give this a
> try. Also, you don't need to completely power off the machine manually
> but just plugging out the network cable works fine. The cloudstack
> agent after losing connection to management server auto reboots
> because of KVM heartbeat check shell script mentioned by Rohit Yadav
> to one of my earlier queries in other thread.
>
> On Mon 12 Mar, 2018, 21:23 Jon Marshall, <jms.123@hotmail.co.uk> wrote:
> Hi Paul
>
>
> Thanks for the response.
>
>
> I think I am not understanding how it was meant to work then. My
> understanding was that the manager used ipmitool to just keep querying
> the compute nodes as to their status so I assumed it didn't matter how
> you shut the node down, once it was down the manager would get no
> response and mark it as down (which it does).
>
>
> I am in testing mode so I think I will just go and pull the power and
> see what happens :)
>
>
> Thanks
>
>
> Jon
>
>
> ________________________________
> From: Paul Angus <paul.angus@shapeblue.com>
> Sent: 12 March 2018 15:31
> To: users@cloudstack.apache.org
> Subject: RE: KVM HostHA
> Hi Jon,
>
> I think that what you guys are finding, is that a controlled host
> shutdown, which will cause the agent to shutdown cleanly; Is not
> considered an HA event. I wouldn't expect CloudStack to take any
> action if you shut down a host, only if the host (agent) stops responding.
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png

[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png

[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

]

]<
> http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



>
> Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



> www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed
by ShapeBlue to deliver the rapid deployment of a standardised ...



> Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge
> is a framework developed by ShapeBlue to deliver the rapid deployment
> of a standardised ...
>
>
>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Jon Marshall <jms.123@hotmail.co.uk>
> Sent: 12 March 2018 15:15
> To: users@cloudstack.apache.org
> Subject: Re: KVM HostHA
>
> I have the same issue here and am not entirely sure what the behaviour
> should be.
>
>
> I have one manager node and 2 compute nodes running 4.11 with ipmi working
> correctly.
>
>
> From the UI under HA -
>
>
> HA Enabled Yes
> HA State Available
> HA Provider kvmhaprovider
>
>
> although interestingly from the "Details" tab it shows -
>
>
> HA enabled No
>
>
> which I assume is a cosmetic issue ?
>
>
> On each compute node I have one HA enabled VM and one non HA enabled VM.
>
>
> I power off a compute node and the UI updates the host status and the VMs
> on that node stop responding but they never fail over to the other node.
>
>
> Couple of things I noticed -
>
>
> 1) as soon as i power off the compute node the HA state on the other node
> shows "Ineligible"
>
>
> 2) In the UI the instances all still show as green even though two of them
> are not available
>
>
> Any help much appreciated
>
>
>
>
> ________________________________
> From: victor <victor@ihnetworks.com>
> Sent: 07 March 2018 17:01
> To: users@cloudstack.apache.org
> Subject: KVM HostHA
>
> Hello Guys,
>
> I have installed cloudstack 4.11. I have enabled HA for each hosts I have
> added. I have also added ipmi successfully (using ipmi driver).
> The hosts are showing like the following.
>
> =======
>
> HA Enabled Yes
> HA State Available
> HA Provider kvmhaprovider
>
> ======
>
> Also the host is showing the following correctly
>
> Resource state --> Enabled
> State --> UP
> Power state --> On
>
> So I have shutdown one of the hosts to see how the KVM hosts Ha is
> working. I have waited for half an hour. But nothing has happened. What
> will happen to the VM's in that host, if the host failed to back up.
> There isn't much from logs.
>
> Regards
> Victor
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message