cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Green <>
Subject Re: kvm live volume migration
Date Fri, 19 Jan 2018 17:45:33 GMT
KVM is able to live migrate entire virtual machines complete with local volumes (see 'man virsh')
but does require nbd (Network Block Device) to be installed on the destination host to do
so. It may need installation of later libvirt / qemu packages from OpenStack repositories
on Centos 6, I'm not sure, but just works on Centos 7. In any event, I have used this functionality
to move virtual machines between virtualization hosts on my home network. It works.

What is missing is the ability to live-migrate a disk from one shared storage to another.
The functionality built into virsh live-migrates the volume ***to the exact same location
on the new host***, so obviously is useless for migrating the disk to a new location on shared
storage. I looked everywhere for the ability of KVM to live migrate a disk from point A to
point B all by itself, and found no such thing. libvirt/qemu has the raw capabilities needed
to do this, but it is not currently exposed as a single API via the qemu console or virsh.
It can be emulated via scripting however:

1. Pause virtual machine
2. Do qcow2 snapshot.
3. Detach base disk, attach qcow2 snapshot
4. unpause virtual machine
5. copy qcow2 base file to new location
6. pause virtual machine
7. detach snapshot
8. unsnapshot qcow2 snapshot at its new location.
9. attach new base at new location.
10. unpause virtual machine.

Thing is, if that entire process is not built into the underlying kvm/qemu/libvirt infrastructure
as tested functionality with a defined API, there's no guarantee that it will work seamlessly
and will continue working with the next release of the underlying infrastructure. This is
using multiple different tools to manipulate the qcow2 file and attach/detach base disks to
the running (but paused) kvm domain, and would have to be tested against all variations of
those tools on all supported Cloudstack KVM host platforms. The test matrix looks pretty grim.

By contrast, the migrate-with-local-storage process is built into virsh and is tested by the
distribution vendor and the set of tools provided with the distribution is guaranteed to work
with the virsh / libvirt/ qemu distributed by the distribution vendor. That makes the test
matrix for move-with-local-storage look a lot simpler -- "is this functionality supported
by that version of virsh on that distribution? Yes? Enable it. No? Don't enable it." 

I'd love to have live migration of disks on shared storage with Cloudstack KVM, but not at
the expense of reliability. Shutting down a virtual machine in order to migrate one of its
disks from one shared datastore to another is not ideal, but at least it's guaranteed reliable.

> On Jan 19, 2018, at 04:54, Rafael Weingärtner <> wrote:
> Hey Marc,
> It is very interesting that you are going to pick this up for KVM. I am
> working in a related issue for XenServer [1].
> If you can confirm that KVM is able to live migrate local volumes to other
> local storage or shared storage I could make the feature I am working on
> available to KVM as well.
> [1]
> On Thu, Jan 18, 2018 at 11:35 AM, Marc-Aurèle Brothier <>
> wrote:
>> There's a PR waiting to be fixed about live migration with local volume for
>> KVM. So it will come at some point. I'm the one who made this PR but I'm
>> not using the upstream release so it's hard for me to debug the problem.
>> You can add yourself to the PR to get notify when things are moving on it.
>> On Wed, Jan 17, 2018 at 10:56 AM, Eric Green <>
>> wrote:
>>> Theoretically on Centos 7 as the host KVM OS it could be done with a
>>> couple of pauses and the snapshotting mechanism built into qcow2, but
>> there
>>> is no simple way to do it directly via virsh, the libvirtd/qemu control
>>> program that is used to manage virtualization. It's not as with issuing a
>>> simple vmotion 'migrate volume' call in Vmware.
>>> I scripted out how it would work without that direct support in
>>> libvirt/virsh and after looking at all the points where things could go
>>> wrong, honestly, I think we need to wait until there is support in
>>> libvirt/virsh to do this. virsh clearly has the capability internally to
>> do
>>> live migration of storage, since it does this for live domain migration
>> of
>>> local storage between machines when migrating KVM domains from one host
>> to
>>> another, but that capability is not currently exposed in a way Cloudstack
>>> could use, at least not on Centos 7.
>>>> On Jan 17, 2018, at 01:05, Piotr Pisz <> wrote:
>>>> Hello,
>>>> Is there a chance that one day it will be possible to migrate volume
>>> (root disk) of a live VM in KVM between storage pools (in CloudStack)?
>>>> Like a storage vMotion in Vmware.
>>>> Best regards,
>>>> Piotr
> -- 
> Rafael Weingärtner

View raw message