cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Turner <>
Subject RE: [DISCUSS] XenServer and HA: the way forward
Date Thu, 28 May 2015 15:45:26 GMT
Thanks, Remi. I wasn't personally involved in the case that went wrong (and that caused Anthony
to rethink this), and I couldn't find the original customer ticket, so I only remember some
of the details. I do remember that the previous CloudStack HA used Pool.emergency_transition_to_master(),
which reverted to a version of the xapi database that was a few minutes out of date. Because
the customer had been making configuration changes during that time (networking changes, I
think), their VMs ended up in a different state than was represented in the database and everything
went wrong from there.

As a first step, as you correctly say, we should document this better in the CloudStack documentation,
and we should also describe the HA timeout in the XenServer documentation. We have filed a
Critical Jira ticket to fix the latter.

As for whether it can be rewritten better, I'm not the best person to ask, but I guess I do
have one point to highlight. It's attractive architecturally for CloudStack to be "in control",
but sometimes XenServer is in a better position to do some things. One of those may be deducing
whether a host is down, where (i) we're within the same cluster, rather than trying to look
from outside; (ii) we have a mature algorithm using redundant channels of handshaking (heartbeats
over the network and heartbeats over storage). So there's a tension there, and I'm not sure
how to resolve it. To what extent should CloudStack try and abstract away the hypervisor,
and to what extent should it use code that is different for each hypervisor but may have advantages
over more generic code? I guess I tend towards the latter, but that may be because I've worked
more in the hypervisor than in the orchestration layer.

Stephen Turner

-----Original Message-----
From: Remi Bergsma [] 
Sent: 28 May 2015 12:53
Subject: Re: [DISCUSS] XenServer and HA: the way forward

Hi Stephen,

Thanks for getting in touch!

First of all, good to hear we agree that this change wasn't done in a nice way. For now, I
think we should focus on documenting this change properly so folks are aware they need to
change their setup. I wrote some documentation already that I'll port to 4.4 and 4.5 as well.
Apart from that, it's not only "turning on HA" since many operations will not work as expected
with HA on.

As for turning on HA by CloudStack, this is not as easy as it sounds. For HA to be enabled
needs to have a SR (storage repository) that the heartbeat can be written to. We use NFS so
we reused that, but if one happens to use iSCSI or Fibre Channel then you need a new SR to
create. We could add warnings indeed, when HA is turned off. Then there is the case with older
versions that require licenses for HA to work. How do you suggest we handle those?

Are there any other things I missed? So far this has been a big "reverse engineering" experience
so it'd be nice if you guys could confirm we've got it all covered now. Also, it'd appreciate
it if you could share some info on the problems that you encountered so we can learn from
it and understand it better.

Final question: can you please take care of documenting the timeout setting in the XenServer
docs? It's a shame it is undocumented.

Looking forward to work together and make this work well again!


2015-05-27 11:45 GMT+02:00 Stephen Turner <>:

> I'm sorry to come late to this thread, but I only picked it up from 
> Remi's blog post [*] over the weekend.
> I'm certainly not going to defend the way this change came in under 
> the radar, but speaking as a member of the XenServer development team, 
> I wouldn't want to go back to the old behaviour. The risk is not just
> theoretical: we had at least one customer with serious data corruption 
> problems as a result of the bad interaction between the CloudStack 
> code and XenServer. I wonder if there's an alternative possibility 
> where CloudStack makes sure that XenServer HA is turned on, and turns 
> it on itself / gives you warnings if it isn't / something?
> --
> Stephen Turner
> [*]
> -sing-and-dance-together-again/
> -----Original Message-----
> From: Remi Bergsma []
> Sent: 04 May 2015 11:04
> To:
> Subject: [DISCUSS] XenServer and HA: the way forward
> Hi all,
> Since CloudStack 4.4 the implementation of HA in CloudStack was 
> changed to use the XenHA feature of XenServer. As of 4.4, it is 
> expected to have XenHA enabled for the pool (not for the VMs!) and so 
> XenServer will be the one to elect a new pool master, whereas 
> CloudStack did it before. Also, XenHA takes care of fencing the box 
> instead of CloudStack should storage be unavailable. To be exact, they 
> both try to fence but XenHA is usually faster.
> To be 100% clear: HA on VMs is in all cases done by CloudStack. It's 
> just that without a pool master, no VMs will be recovered anyway. This 
> brought some headaches to me, as first of all I didn't know. We 
> probably need to document this somewhere. This is important, because 
> without XenHA turned on you'll not get a new pool master (a behaviour change).
> Personally, I don't like the fact that we have "two captains" in case 
> something goes wrong. But, some say they like this behaviour. I'm OK 
> with both, as long as one can choose whatever suits their needs best.
> In Austin I talked to several people about this. We came up with the 
> idea to have CloudStack check whether XenHA is on or not. If it is, it 
> does the current 4.4+ behaviour (XenHA selects new pool master). When 
> it is not, we do the CloudStack 4.3 behaviour where CloudStack is fully in control.
> I also talked to Tim Mackey and he wants to help implement this, but 
> he doesn't have much time. The idea is to have someone else join in to 
> code the change and then Tim will be able to help out on a regularly 
> basis should we need in depth knowledge of XenServer or its 
> implementation in CloudStack.
> Before we kick this off, I'd like to discuss and agree that this is 
> the way forward. Also, if you're interested in joining this effort let 
> me know and I'll kick it off.
> Regards,
> Remi
View raw message