cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koushik Das <>
Subject RE: [DISCUSS] Out of Band VR migration, should we reboot VR or not?
Date Fri, 05 Jun 2015 07:29:49 GMT
From the persistent config changes, it appears that aggregate command approach is not required
in 4.6 and onwards. If someone wants to implement the aggregate approach only for 4.4 and
4.5 irrespective of the effort involved then that should be fine.

Otherwise we can just put some config to decide between #1 or #2. Another alternate could
be to replace the reboot logic with an alert if there is an out of band VR migration. Based
on it admin can reboot the VR from CCP if required. There will be a window during which the
VR will be without ant config but that can be documented.

-----Original Message-----
From: Remi Bergsma [] 
Sent: Thursday, 4 June 2015 19:56
Subject: Re: [DISCUSS] Out of Band VR migration, should we reboot VR or not?


On 04 Jun 2015, at 14:29, Rohit Yadav <<>>


On 04-Jun-2015, at 11:05 am, Remi Bergsma <<>>

To summarise:
#1. rebooting VR is needed for hypervisors that have their own DR (like VMware and Hyperv)
as a restart outside of CloudStack makes it lose its config hence the VR is unavailable #2.
rebooting is NOT needed for successful live migrations on _any_ hypervisor (since there was
no restart everything still works) #3. CloudStack 4.6 has persistent config in VR, so rebooting
is never needed

The current behaviour in 4.4.3, 4.5.1 and master:
- always rebooting VR when out-of-band detected ==> Works great for #1, but makes case
#2 not work

The previous behaviour:
- never rebooting VR when out-of-band detected ==>  Works great for #2, but makes case
#1 not work

We need something that works for both cases :-)

About #1 and #2:
Can we detect in another way that a VR became unreachable/non-functional and do an action
based on that?

So, if a VR lives on a VMware hypervisor that happens to crash, VMware HA will start it on
another available hypervisor but without config it will not be reachable on the control network.
If we want to do it generic, I’d say that when a VR is not controllable any more we could
reboot it. We could also make this a setting ‘systemvm auto reboot on control failure’
or whatever we call it.

This would then also be a useful feature in 4.6

About #3:
I’d suggest at least reverting the commit to master, as it makes no sense since the VR is
persistent already.;a=commitdiff;h=c3515c9

I agree on master, we can revert due to #3.

On 4.4 and 4.5 branches, if we revert it than it will break #1 and if we keep it - it breaks
#2. We need to fix this in a hypervisor agnostic way.

So, if we can use aggregated cmd execution that Sudhashu and others have shared, in case of
an out of band migration we can do this:

1. If we’re unable to reach VR, we can keep the RebootTask to do its job. In many cases
I’ve seen VR disk corruption, so if after rebooting VR (which might kick in fsck) it fails
CloudStack may eventually recreate a new VR makes sense. This will solve #1. This case IMO
is highly unlikely.

2. If we’re able to reach the VR after VR is migrated out of band, we run an AgrregationExecutionTask
sending all the rules for that VR. This will solve #2 without needing to reboot a VM. This
case is more likely to happen, so if I’ve pick between this and the above case, I would
try to solve for this case.

I’m not sure about the actual implementation, and will re-applying rules using a AgrregationExecutionTask
cause any issue in the VR? Comments?

Rohit Yadav
Software Architect, ShapeBlue
M. +91 88 262 30892 |<>
Blog:<> | Twitter: @_bhaisaab

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<>
CSForge – rapid IaaS deployment framework<>
CloudStack Consulting<>
CloudStack Software Engineering<>
CloudStack Infrastructure Support<>
CloudStack Bootcamp Training Courses<>

This email and any attachments to it may be confidential and are intended solely for the use
of the individual to whom it is addressed. Any views or opinions expressed are solely those
of the author and do not necessarily represent those of Shape Blue Ltd or related companies.
If you are not the intended recipient of this email, you must neither take any action based
upon its contents, nor copy or show it to anyone. Please contact the sender if you believe
you have received this email in error. Shape Blue Ltd is a company incorporated in England
& Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
registered by The Republic of South Africa and is traded under license from Shape Blue Ltd.
ShapeBlue is a registered trademark.

View raw message