cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yoshikazu Nojima <m...@ynojima.net>
Subject Re: [RFC] adding volume provisioning method option
Date Sat, 01 Feb 2014 07:01:28 GMT
Yes, preallocation=metadata creates a large file full of holes. and
maybe I need to modify storage consumption measuring method of
CloudStack to care this kind sparse file.

To satisfy my needs, just turning on preallocation=metadata by default
or making it configurable from agent.properties is enough, but I'm
still thinking about making it configurable from disk offerings.

Thanks,

2014-01-31 Marcus <shadowsor@gmail.com>:
> Ok, so not technically sparse in the sense that you have a large file
> full of holes, but it's still not allocating the entire disk upfront.
> You get the same result of disk savings either way, existing
> cloudstack installs with qcow2 are still saving space in the same
> manner.
>
> On Fri, Jan 31, 2014 at 11:43 PM, Marcus <shadowsor@gmail.com> wrote:
>> Here are my tests, using stock centos 6.5 qemu-img:
>>
>> [root@server ~]# qemu-img create -f qcow2 image.qcow2 10G
>> Formatting 'image.qcow2', fmt=qcow2 size=10737418240 encryption=off
>> cluster_size=65536
>> [root@server ~]# qemu-img info image.qcow2
>> image: image.qcow2
>> file format: qcow2
>> virtual size: 10G (10737418240 bytes)
>> disk size: 136K
>> cluster_size: 65536
>>
>> No options leaves the disk as 136K, it's sparse.
>>
>> [root@server ~]# qemu-img create -f qcow2 -o preallocation=metadata
>> imagepre.qcow2 10G
>> Formatting 'imagepre.qcow2', fmt=qcow2 size=10737418240 encryption=off
>> cluster_size=65536 preallocation='metadata'
>> [root@server ~]# qemu info imagepre.qcow2
>> -bash: qemu: command not found
>> [root@server ~]# qemu-img info imagepre.qcow2
>> image: imagepre.qcow2
>> file format: qcow2
>> virtual size: 10G (10737418240 bytes)
>> disk size: 1.7M
>> cluster_size: 65536
>>
>> preallocation=metadata leaves disk as 1.7M, it's also sparse, but a bit bigger.
>>
>> On Fri, Jan 31, 2014 at 11:41 PM, Marcus <shadowsor@gmail.com> wrote:
>>> Yes, no_option.img is only 256k in size. it's sparse.
>>>
>>> On Fri, Jan 31, 2014 at 11:38 PM, Yoshikazu Nojima <mail@ynojima.net> wrote:
>>>>>I don't think preallocation=metadata is required for sparse qcow2.
>>>>>Just if you want it to NOT be sparse for metadata.
>>>>
>>>> I think it is required. Please see the output below. Only
>>>> "metadata.img" is a sparse file.
>>>>
>>>> ==============
>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>> preallocation=off off.img 3G
>>>> Formatting 'off.img', fmt=qcow2 size=3221225472 encryption=off
>>>> cluster_size=65536 preallocation='off'
>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>> preallocation=metadata metadata.img 3G
>>>> Formatting 'metadata.img', fmt=qcow2 size=3221225472 encryption=off
>>>> cluster_size=65536 preallocation='metadata'
>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>> preallocation=full full.img 3G
>>>> Formatting 'full.img', fmt=qcow2 size=3221225472 encryption=off
>>>> cluster_size=65536 preallocation='full'
>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 no_option.img 3G
>>>> Formatting 'no_option.img', fmt=qcow2 size=3221225472 encryption=off
>>>> cluster_size=65536
>>>> [root@ut1-qmb-rd5c test]# ls -alhs
>>>> total 3.1G
>>>> 4.0K drwxr-xr-x  2 root root 4.0K Feb  1 06:29 .
>>>> 4.0K dr-xr-x---. 5 root root 4.0K Feb  1 06:19 ..
>>>> 3.1G -rw-r--r--  1 root root 3.1G Feb  1 06:23 full.img
>>>> 592K -rw-r--r--  1 root root 3.1G Feb  1 06:21 metadata.img
>>>> 136K -rw-r--r--  1 root root 256K Feb  1 06:29 no_option.img
>>>> 136K -rw-r--r--  1 root root 256K Feb  1 06:21 off.img
>>>> ==============
>>>>
>>>>
>>>>>I also remember hearing that either recent qemu-img has a decent
>>>>>preallocation by default now, or that performance has improved such
>>>>>that it doesn't matter. Will need to do some reading to remember, but
>>>>>I think I remember hearing it's not a big deal nowadays.
>>>>
>>>> Ummm. I tested with "qemu-img version 0.12.1," and I confirmed the
>>>> performance of "preallocation=off" is worse than
>>>> "preallocation=metadata",
>>>> but it may be not true with latest qemu. I will test it.
>>>>
>>>> Thank you for your comment!
>>>>
>>>> Regards,
>>>>
>>>>
>>>> 2014-01-31 Marcus <shadowsor@gmail.com>:
>>>>> I don't think preallocation=metadata is required for sparse qcow2.
>>>>> Just if you want it to NOT be sparse for metadata.
>>>>>
>>>>> I also remember hearing that either recent qemu-img has a decent
>>>>> preallocation by default now, or that performance has improved such
>>>>> that it doesn't matter. Will need to do some reading to remember, but
>>>>> I think I remember hearing it's not a big deal nowadays.
>>>>>
>>>>> On Fri, Jan 31, 2014 at 10:04 PM, Yoshikazu Nojima <mail@ynojima.net>
wrote:
>>>>>> Hello Nux,
>>>>>>
>>>>>> Thank you for your comment. I will prepare feature specification.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> 2014-01-31 Nux! <nux@li.nux.ro>:
>>>>>>> On 31.01.2014 20:24, Yoshikazu Nojima wrote:
>>>>>>>>
>>>>>>>> Afternoon All,
>>>>>>>>
>>>>>>>> Is there anyone working on adding volume provisioning method
option?
>>>>>>>> As you know, thin provisioning of a volume save consumption
of a
>>>>>>>> storage, and fat provisioning improves IOPS performance.
>>>>>>>> Especially, Qcow2 can save storage consumption and achive
relatively
>>>>>>>> better performance than default by provisioning a volume
with an
>>>>>>>> option "preallocation=metadata", which makes an image file
a sparse
>>>>>>>> file.
>>>>>>>>
>>>>>>>> http://itscblog.tamu.edu/improve-disk-io-performance-in-kvm/#.232---preallocation
>>>>>>>>
>>>>>>>> Any thoughts about this?
>>>>>>>> If it is ok, I will write a feature specification on confluence
and
>>>>>>>> start implementation.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Noji
>>>>>>>>
>>>>>>>>
>>>>>>>> Yoshikazu Nojima <mail@ynojima.net>
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I thought preallocation=metadata is common practice since years.
Now I find
>>>>>>> out ACS doesn't actually use it?
>>>>>>> If so, this is really _bad_ and needs to be fixed ASAP...
>>>>>>>
>>>>>>> Thanks Noji
>>>>>>>
>>>>>>> --
>>>>>>> Sent from the Delta quadrant using Borg technology!
>>>>>>>
>>>>>>> Nux!
>>>>>>> www.nux.ro

Mime
View raw message