cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wido den Hollander <>
Subject Re: ACS and KVM uses /tmp for volumes migration and templates
Date Thu, 20 Mar 2014 09:40:29 GMT
On 03/20/2014 12:59 AM, Andrei Mikhailovsky wrote:
> Hi guys,
> I was wondering if this is a bug?

No, it's a "feature".

> I've noticed that during volume migration from NFS to RBD primary storage the volume
image is first copied to /tmp and only then to the RBD storage. This seems silly to me as
one would expect a typical volume to be larger than the host's hard disk. Also, it is a common
practice to use tmpfs as /tmp for performance reasons. Thus, a typical host server will have
far smaller /tmp folder than the size of an average volume. As a result, volume migration
would break after filling the /tmp and could probably cause a bunch of issue for the KVM host
itself as well as any vms running on the server.

Correct. The problem was that RBD images know two formats. Format 1 
(old/legacy) and format 2.

In order to perform cloning images should be in RBD format 2.

When running qemu-img convert with a RBD image as a destination qemu-img 
will create a RBD image in format 1.

That's due to this piece of code in block/rbd.c in Qemu:

     ret = rbd_create(io_ctx, name, bytes, &obj_order);

rbd_create() creates images in format 1. To use format 2 you should use 
rbd_create2() or rbd_create3().

With RBD format 1 we can't do snapshotting or cloning, which we require 
in ACS.

So I had to do a intermediate step where I first wrote the RAW image 
somewhere and afterwards write it to RBD.

After some discussion a config option has been added to Ceph:

OPTION(rbd_default_format, OPT_INT, 1)

This allows me to do this:

qemu-img convert .. -O raw .. rbd:rbd/myimage:rbd_default_format=2

This causes librbd/RBD to create a format 2 image and we can skip the 
convert step to /tmp.

This option is available since Ceph Dumpling 0.67.5 and was not 
available when ACS 4.2 was written.

I'm going to make changes in master which skip the step with /tmp.

Technically this can be backported to 4.2, but then you would have to 
run your own homebrew version of 4.2

> It also seems that the /tmp is temporarily used during a template creation .

Same story as above.

> My setup:
> ACS 4.2.1
> Ubuntu 12.04 with KVM
> RBD + NFS for Primary storage
> NFS for Staging and Secondary storage
> Thanks
> Andrei

View raw message