cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus <shadow...@gmail.com>
Subject Re: [RFC] adding volume provisioning method option
Date Sat, 01 Feb 2014 07:12:24 GMT
I don't think you'd have to change anything. Cloudstack (at least for
accounting purposes) considers all storage 'fat' and calculates
capacity per the disk offering size compared to total size of the
primary pool. Then you can add an overprovisioning factor to fit your
needs in the config options.

On Sat, Feb 1, 2014 at 12:01 AM, Yoshikazu Nojima <mail@ynojima.net> wrote:
> Yes, preallocation=metadata creates a large file full of holes. and
> maybe I need to modify storage consumption measuring method of
> CloudStack to care this kind sparse file.
>
> To satisfy my needs, just turning on preallocation=metadata by default
> or making it configurable from agent.properties is enough, but I'm
> still thinking about making it configurable from disk offerings.
>
> Thanks,
>
> 2014-01-31 Marcus <shadowsor@gmail.com>:
>> Ok, so not technically sparse in the sense that you have a large file
>> full of holes, but it's still not allocating the entire disk upfront.
>> You get the same result of disk savings either way, existing
>> cloudstack installs with qcow2 are still saving space in the same
>> manner.
>>
>> On Fri, Jan 31, 2014 at 11:43 PM, Marcus <shadowsor@gmail.com> wrote:
>>> Here are my tests, using stock centos 6.5 qemu-img:
>>>
>>> [root@server ~]# qemu-img create -f qcow2 image.qcow2 10G
>>> Formatting 'image.qcow2', fmt=qcow2 size=10737418240 encryption=off
>>> cluster_size=65536
>>> [root@server ~]# qemu-img info image.qcow2
>>> image: image.qcow2
>>> file format: qcow2
>>> virtual size: 10G (10737418240 bytes)
>>> disk size: 136K
>>> cluster_size: 65536
>>>
>>> No options leaves the disk as 136K, it's sparse.
>>>
>>> [root@server ~]# qemu-img create -f qcow2 -o preallocation=metadata
>>> imagepre.qcow2 10G
>>> Formatting 'imagepre.qcow2', fmt=qcow2 size=10737418240 encryption=off
>>> cluster_size=65536 preallocation='metadata'
>>> [root@server ~]# qemu info imagepre.qcow2
>>> -bash: qemu: command not found
>>> [root@server ~]# qemu-img info imagepre.qcow2
>>> image: imagepre.qcow2
>>> file format: qcow2
>>> virtual size: 10G (10737418240 bytes)
>>> disk size: 1.7M
>>> cluster_size: 65536
>>>
>>> preallocation=metadata leaves disk as 1.7M, it's also sparse, but a bit bigger.
>>>
>>> On Fri, Jan 31, 2014 at 11:41 PM, Marcus <shadowsor@gmail.com> wrote:
>>>> Yes, no_option.img is only 256k in size. it's sparse.
>>>>
>>>> On Fri, Jan 31, 2014 at 11:38 PM, Yoshikazu Nojima <mail@ynojima.net>
wrote:
>>>>>>I don't think preallocation=metadata is required for sparse qcow2.
>>>>>>Just if you want it to NOT be sparse for metadata.
>>>>>
>>>>> I think it is required. Please see the output below. Only
>>>>> "metadata.img" is a sparse file.
>>>>>
>>>>> ==============
>>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>>> preallocation=off off.img 3G
>>>>> Formatting 'off.img', fmt=qcow2 size=3221225472 encryption=off
>>>>> cluster_size=65536 preallocation='off'
>>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>>> preallocation=metadata metadata.img 3G
>>>>> Formatting 'metadata.img', fmt=qcow2 size=3221225472 encryption=off
>>>>> cluster_size=65536 preallocation='metadata'
>>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 -o
>>>>> preallocation=full full.img 3G
>>>>> Formatting 'full.img', fmt=qcow2 size=3221225472 encryption=off
>>>>> cluster_size=65536 preallocation='full'
>>>>> [root@ut1-qmb-rd5c test]# qemu-img create -f qcow2 no_option.img 3G
>>>>> Formatting 'no_option.img', fmt=qcow2 size=3221225472 encryption=off
>>>>> cluster_size=65536
>>>>> [root@ut1-qmb-rd5c test]# ls -alhs
>>>>> total 3.1G
>>>>> 4.0K drwxr-xr-x  2 root root 4.0K Feb  1 06:29 .
>>>>> 4.0K dr-xr-x---. 5 root root 4.0K Feb  1 06:19 ..
>>>>> 3.1G -rw-r--r--  1 root root 3.1G Feb  1 06:23 full.img
>>>>> 592K -rw-r--r--  1 root root 3.1G Feb  1 06:21 metadata.img
>>>>> 136K -rw-r--r--  1 root root 256K Feb  1 06:29 no_option.img
>>>>> 136K -rw-r--r--  1 root root 256K Feb  1 06:21 off.img
>>>>> ==============
>>>>>
>>>>>
>>>>>>I also remember hearing that either recent qemu-img has a decent
>>>>>>preallocation by default now, or that performance has improved such
>>>>>>that it doesn't matter. Will need to do some reading to remember,
but
>>>>>>I think I remember hearing it's not a big deal nowadays.
>>>>>
>>>>> Ummm. I tested with "qemu-img version 0.12.1," and I confirmed the
>>>>> performance of "preallocation=off" is worse than
>>>>> "preallocation=metadata",
>>>>> but it may be not true with latest qemu. I will test it.
>>>>>
>>>>> Thank you for your comment!
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> 2014-01-31 Marcus <shadowsor@gmail.com>:
>>>>>> I don't think preallocation=metadata is required for sparse qcow2.
>>>>>> Just if you want it to NOT be sparse for metadata.
>>>>>>
>>>>>> I also remember hearing that either recent qemu-img has a decent
>>>>>> preallocation by default now, or that performance has improved such
>>>>>> that it doesn't matter. Will need to do some reading to remember,
but
>>>>>> I think I remember hearing it's not a big deal nowadays.
>>>>>>
>>>>>> On Fri, Jan 31, 2014 at 10:04 PM, Yoshikazu Nojima <mail@ynojima.net>
wrote:
>>>>>>> Hello Nux,
>>>>>>>
>>>>>>> Thank you for your comment. I will prepare feature specification.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> 2014-01-31 Nux! <nux@li.nux.ro>:
>>>>>>>> On 31.01.2014 20:24, Yoshikazu Nojima wrote:
>>>>>>>>>
>>>>>>>>> Afternoon All,
>>>>>>>>>
>>>>>>>>> Is there anyone working on adding volume provisioning
method option?
>>>>>>>>> As you know, thin provisioning of a volume save consumption
of a
>>>>>>>>> storage, and fat provisioning improves IOPS performance.
>>>>>>>>> Especially, Qcow2 can save storage consumption and achive
relatively
>>>>>>>>> better performance than default by provisioning a volume
with an
>>>>>>>>> option "preallocation=metadata", which makes an image
file a sparse
>>>>>>>>> file.
>>>>>>>>>
>>>>>>>>> http://itscblog.tamu.edu/improve-disk-io-performance-in-kvm/#.232---preallocation
>>>>>>>>>
>>>>>>>>> Any thoughts about this?
>>>>>>>>> If it is ok, I will write a feature specification on
confluence and
>>>>>>>>> start implementation.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Noji
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yoshikazu Nojima <mail@ynojima.net>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I thought preallocation=metadata is common practice since
years. Now I find
>>>>>>>> out ACS doesn't actually use it?
>>>>>>>> If so, this is really _bad_ and needs to be fixed ASAP...
>>>>>>>>
>>>>>>>> Thanks Noji
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from the Delta quadrant using Borg technology!
>>>>>>>>
>>>>>>>> Nux!
>>>>>>>> www.nux.ro

Mime
View raw message