cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nux! <...@li.nux.ro>
Subject Re: Poor NVMe Performance with KVM
Date Fri, 17 May 2019 13:20:57 GMT
What happens when you set deadline scheduler in both HV and guest?

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Ivan Kudryavtsev" <kudryavtsev_ia@bw-sw.com>
> To: "users" <users@cloudstack.apache.org>, "dev" <dev@cloudstack.apache.org>
> Sent: Friday, 17 May, 2019 14:16:31
> Subject: Re: Poor NVMe Performance with KVM

> BTW, You may think that the improvement is achieved by caching, but I clear
> the cache with
> sync & echo 3 > /proc/sys/vm/drop_caches
> 
> So, can't claim for sure, need other opinion, but looks like for NVMe,
> writethrough must be used if you want high IO rate. At least with Intel
> p4500.
> 
> 
> пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>:
> 
>> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
>> directly in DB and the performance boosted greatly. It may be an important
>> feature for NVME drives.
>>
>> Currently, on 4.11, the user can set cache-mode for disk offerings, but
>> cannot for service offerings, which are translated to cache=none
>> corresponding disk offerings.
>>
>> The only way is to use SQL to change that for root disk disk offerings.
>> CreateServiceOffering API doesn't support cache mode. It can be a serious
>> limitation for NVME users, because by default they could meet poor
>> read/write performance.
>>
>> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>:
>>
>>> Darius, thanks for your participation,
>>>
>>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>>> switched to 4.15 with dist-upgrade.
>>>
>>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>>> Cloudstack?
>>>
>>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <daznis@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I can see a few issues with your xml file. You can try using "queues"
>>>> inside your disk definitions. This should help a little, not sure by
>>>> how much for your case, but for my specific it went up by almost the
>>>> number of queues. Also try cache directsync or writethrough. You
>>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>>
>>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>>> <kudryavtsev_ia@bw-sw.com> wrote:
>>>> >
>>>> > Hello, colleagues.
>>>> >
>>>> > Hope, someone could help me. I just deployed a new VM host with Intel
>>>> P4500
>>>> > local storage NVMe drive.
>>>> >
>>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>>> with
>>>> > FIO, write performance is also high as expected.
>>>> >
>>>> > I've created a new KVM VM Service offering with virtio-scsi controller
>>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>>> with
>>>> > FIO. Results are very strange:
>>>> >
>>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>>>> limits
>>>> > are R=1000/W=500 MBs).
>>>> >
>>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>>> > direct=1 leads to very moderate 2-3K IOPS.
>>>> >
>>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>>> >
>>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
>>>> which is
>>>> > strange.
>>>> >
>>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>>>> are
>>>> > executed. From the host, it works great.
>>>> >
>>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>>> > performance is nice. Maybe someone managed to use NVME with KVM with
>>>> small
>>>> > IOPS?
>>>> >
>>>> > The filesystem is XFS, previously tried with EXT4 - results are the
>>>> same.
>>>> >
>>>> > This is the part of VM XML definition generated by CloudStack:
>>>> >
>>>> >   <devices>
>>>> >     <emulator>/usr/bin/kvm-spice</emulator>
>>>> >     <disk type='file' device='disk'>
>>>> >       <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>>> >       <source
>>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>>> >       <backingStore type='file' index='1'>
>>>> >         <format type='raw'/>
>>>> >         <source
>>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>>> >         <backingStore/>
>>>> >       </backingStore>
>>>> >       <target dev='sda' bus='scsi'/>
>>>> >       <iotune>
>>>> >         <read_bytes_sec>1048576000</read_bytes_sec>
>>>> >         <write_bytes_sec>524288000</write_bytes_sec>
>>>> >         <read_iops_sec>100000</read_iops_sec>
>>>> >         <write_iops_sec>50000</write_iops_sec>
>>>> >       </iotune>
>>>> >       <serial>6809dbd04a1540149322</serial>
>>>> >       <alias name='scsi0-0-0-0'/>
>>>> >       <address type='drive' controller='0' bus='0' target='0'
>>>> unit='0'/>
>>>> >     </disk>
>>>> >     <disk type='file' device='cdrom'>
>>>> >       <driver name='qemu' type='raw'/>
>>>> >       <backingStore/>
>>>> >       <target dev='hdc' bus='ide'/>
>>>> >       <readonly/>
>>>> >       <alias name='ide0-1-0'/>
>>>> >       <address type='drive' controller='0' bus='1' target='0'
>>>> unit='0'/>
>>>> >     </disk>
>>>> >     <controller type='scsi' index='0' model='virtio-scsi'>
>>>> >       <alias name='scsi0'/>
>>>> >       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>>> > function='0x0'/>
>>>> >     </controller>
>>>> >
>>>> > So, what I see now, is that it works slower than couple of two Samsung
>>>> 960
>>>> > PRO which is extremely strange.
>>>> >
>>>> > Thanks in advance.
>>>> >
>>>> >
>>>> > --
>>>> > With best regards, Ivan Kudryavtsev
>>>> > Bitworks LLC
>>>> > Cell RU: +7-923-414-1515
>>>> > Cell USA: +1-201-257-1512
>>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>>

Mime
View raw message