cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: Caching modes
Date Tue, 20 Feb 2018 20:02:19 GMT
Thanks, we will proceed reviweing

On Tue, Feb 20, 2018 at 3:12 PM, Andrija Panic <andrija.panic@gmail.com>
wrote:

> Here it is:
>
> https://github.com/apache/cloudstack-docs-admin/pull/47
>
> Added KVM online storage migration (atm only CEPH/NFS to SolidFire, new in
> 4.11 release)
> Added KVM cache mode setup and limitations.
>
>
> Cheers
>
> On 20 February 2018 at 16:49, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
> > If you are willing to write it down, please do so, and open a PR. We will
> > review and merged it afterwards.
> >
> > On Tue, Feb 20, 2018 at 12:41 PM, Andrija Panic <andrija.panic@gmail.com
> >
> > wrote:
> >
> > > I advise (or not...depends on point of view) to stay that way - because
> > > when you activate write-back cache - live migrations will stop, and
> this
> > > makes *Enable maintenance mode (put host into maintenance)* impossible.
> > >
> > > I would perhaps suggest that there is documentation for "advanced
> users"
> > or
> > > similar, that will say "it's possible to enable this and this way via
> DB
> > > hack, but be warned on live migration consequences etc..." since this
> > will
> > > render more problems if people start using it.
> > >
> > > If you choose to do, let me know, I can write that (documentation)
> > briefly.
> > >
> > > Not to mention it can be unsafe (power failure - less possible I guess,
> > but
> > > rare kernel panic etc might have it's consequences I assume)
> > >
> > > It does indeed increase performance on NFS much, but not necessarily on
> > > CEPH (if you are using lirbd cache on client side as explained above)
> > >
> > > On 20 February 2018 at 15:48, Rafael Weingärtner <
> > > rafaelweingartner@gmail.com> wrote:
> > >
> > > > Yes. Weird enough, the code is using the value in the database if it
> is
> > > > provided there, but there is no easy way for users to change that
> > > > configuration in the database. ¯\_(ツ)_/¯
> > > >
> > > > On Tue, Feb 20, 2018 at 11:45 AM, Andrija Panic <
> > andrija.panic@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > So it seems that just passing the cachemode value to API is not
> > there,
> > > or
> > > > > somehow messedup, but deployVM process does read DB values from
> > > > > disk_offering table for sure, and applies it to XML file for KVM.
> > > > > This is above ACS 4.8.x.
> > > > >
> > > > >
> > > > > On 20 February 2018 at 15:44, Andrija Panic <
> andrija.panic@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I have edited the disk_offering table, in the cache_mode just
> enter
> > > > > > "writeback". Stop and start VM, and it will pickup/inherit the
> > > > cache_mode
> > > > > > from it's parrent offering
> > > > > > This also applies to Compute/Service offering, again inside
> > > > disk_offering
> > > > > > table - just tested both
> > > > > >
> > > > > > i.e.
> > > > > >
> > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback'
WHERE
> > > > > > `id`=102; # Compute Offering (Service offering)
> > > > > > UPDATE `cloud`.`disk_offering` SET `cache_mode`='writeback'
WHERE
> > > > > > `id`=114; #data disk offering
> > > > > >
> > > > > > Before SQL:
> > > > > >
> > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > >       <driver name='qemu' type='qcow2' cache='none'/>
> > > > > >       <source file='/mnt/63a3ae7b-9ea9-3884-
> > > > a772-1ea939ef6ec3/1b655159-
> > > > > > ae10-41cf-8987-f1cfb47fe453'/>
> > > > > >       <target dev='vda' bus='virtio'/>
> > > > > > --
> > > > > >       <driver name='qemu' type='qcow2' cache='none'/>
> > > > > >       <source file='/mnt/63a3ae7b-9ea9-3884-
> > > > a772-1ea939ef6ec3/09bdadcb-
> > > > > > ec6e-4dda-b37b-17b1a749257f'/>
> > > > > >       <target dev='vdb' bus='virtio'/>
> > > > > > --
> > > > > >
> > > > > > STOP and START VM = after SQL
> > > > > >
> > > > > > root@ix1-c7-4:~# virsh dumpxml i-2-10-VM | grep cache -A2
> > > > > >       <driver name='qemu' type='qcow2' cache='writeback'/>
> > > > > >       <source file='/mnt/63a3ae7b-9ea9-3884-
> > > > a772-1ea939ef6ec3/1b655159-
> > > > > > ae10-41cf-8987-f1cfb47fe453'/>
> > > > > >       <target dev='vda' bus='virtio'/>
> > > > > > --
> > > > > >       <driver name='qemu' type='qcow2' cache='writeback'/>
> > > > > >       <source file='/mnt/63a3ae7b-9ea9-3884-
> > > > a772-1ea939ef6ec3/09bdadcb-
> > > > > > ec6e-4dda-b37b-17b1a749257f'/>
> > > > > >       <target dev='vdb' bus='virtio'/>
> > > > > > --
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 20 February 2018 at 14:03, Rafael Weingärtner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > >> I have no idea how it can change the performance. If you
look at
> > the
> > > > > >> content of the commit you provided, it is only the commit
that
> > > enabled
> > > > > the
> > > > > >> use of getCacheMode from disk offerings. However, it is
not
> > exposing
> > > > any
> > > > > >> way to users to change that value/configuration in the
> database. I
> > > > might
> > > > > >> have missed it; do you see any API methods that receive
the
> > > parameter
> > > > > >> "cacheMode" and then pass this parameter to a "diskOffering"
> > object,
> > > > and
> > > > > >> then persist/update this object in the database?
> > > > > >>
> > > > > >> May I ask how are you guys changing the cacheMode configuration?
> > > > > >>
> > > > > >> On Tue, Feb 20, 2018 at 9:56 AM, Paul Angus <
> > > paul.angus@shapeblue.com
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I'm working with some guys who are experimenting with
the
> > setting
> > > as
> > > > > if
> > > > > >> > definitely seems to change the performance of data
disks.  It
> > also
> > > > > >> changes
> > > > > >> > the XML of the VM which is created.
> > > > > >> >
> > > > > >> > p.s.
> > > > > >> > I've found this commit;
> > > > > >> >
> > > > > >> > https://github.com/apache/cloudstack/commit/1edaa36cc68e845a
> > > > > >> 42339d5f267d49
> > > > > >> > c82343aefb
> > > > > >> >
> > > > > >> > so I've got something to investigate now, but API
> documentation
> > > must
> > > > > >> > definitely be askew.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > paul.angus@shapeblue.com
> > > > > >> > www.shapeblue.com
> > > > > >> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >> > @shapeblue
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > -----Original Message-----
> > > > > >> > From: Rafael Weingärtner [mailto:rafaelweingartner@gmail.com]
> > > > > >> > Sent: 20 February 2018 12:31
> > > > > >> > To: dev <dev@cloudstack.apache.org>
> > > > > >> > Subject: Re: Caching modes
> > > > > >> >
> > > > > >> > This cache mode parameter does not exist in
> > > "CreateDiskOfferingCmd"
> > > > > >> > command. I also checked some commits from 2, 3, 4 and
5 years
> > ago,
> > > > and
> > > > > >> > this parameter was never there. If you check the API
in [1],
> you
> > > can
> > > > > see
> > > > > >> > that it is not an expected parameter. Moreover, I do
not see
> any
> > > use
> > > > > of
> > > > > >> > "setCacheMode" in the code (in case it is updated by
some
> other
> > > > > method).
> > > > > >> > Interestingly enough, the code uses "getCacheMode".
> > > > > >> >
> > > > > >> > In summary, it is not a feature, and it does not work.
It
> looks
> > > like
> > > > > >> some
> > > > > >> > leftover from dark ages when people could commit anything
and
> > then
> > > > > they
> > > > > >> > would just leave a half implementation there in our
code base.
> > > > > >> >
> > > > > >> > [1]
> > > > > >> > https://cloudstack.apache.org/api/apidocs-4.11/apis/
> > > > > >> > createDiskOffering.html
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Feb 20, 2018 at 9:19 AM, Andrija Panic <
> > > > > andrija.panic@gmail.com
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > I can also assume that "cachemode" as API parameter
is not
> > > > > supported,
> > > > > >> > > since when creating data disk offering via GUI
also doesn't
> > set
> > > it
> > > > > in
> > > > > >> > DB/table.
> > > > > >> > >
> > > > > >> > > CM:    create diskoffering name=xxx displaytext=xxx
> > > > > storagetype=shared
> > > > > >> > > disksize=1024 cachemode=writeback
> > > > > >> > >
> > > > > >> > > this also does not set cachemode in table... my
guess it's
> not
> > > > > >> > > implemented in API
> > > > > >> > >
> > > > > >> > > Let me know if I can help with any testing here.
> > > > > >> > >
> > > > > >> > > Cheers
> > > > > >> > >
> > > > > >> > > On 20 February 2018 at 13:09, Andrija Panic <
> > > > > andrija.panic@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Paul,
> > > > > >> > > >
> > > > > >> > > > not helping directly answering your question,
but here are
> > > some
> > > > > >> > > > observations and "warning" if client's are
using
> write-back
> > > > cache
> > > > > on
> > > > > >> > > > KVM level
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > I have (long time ago) tested performance
in 3
> combinations
> > > > (this
> > > > > >> > > > was not really thorough testing but a brief
testing with
> FIO
> > > and
> > > > > >> > > > random IO WRITE)
> > > > > >> > > >
> > > > > >> > > > - just CEPH rbd cache (on KVM side)
> > > > > >> > > >            i.e. [client]
> > > > > >> > > >                  rbd cache = true
> > > > > >> > > >                  rbd cache writethrough until
flush = true
> > > > > >> > > >                  #(this is default 32MB per
volume, afaik
> > > > > >> > > >
> > > > > >> > > > - just KMV write-back cache (had to manually
edit
> > > disk_offering
> > > > > >> > > > table to activate cache mode, since when
creating new disk
> > > > > offering
> > > > > >> > > > via GUI, the disk_offering tables was NOT
populated with
> > > > > >> > > > "write-back" sertting/value
> > > > > >> > > ! )
> > > > > >> > > >
> > > > > >> > > > - both CEPH and KVM write-back cahce active
> > > > > >> > > >
> > > > > >> > > > My observations were like following, but
would be good to
> > > > actually
> > > > > >> > > confirm
> > > > > >> > > > by someone else:
> > > > > >> > > >
> > > > > >> > > > - same performance with only CEPH caching
or with only KVM
> > > > caching
> > > > > >> > > > - a bit worse performance with both CEPH
and KVM caching
> > > active
> > > > > >> > > > (nonsense combination, I know...)
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Please keep in mind, that some ACS functionality,
KVM
> > > > > >> > > > live-migrations on shared storage (NFS/CEPH)
are NOT
> > supported
> > > > > when
> > > > > >> > > > you use KVM write-back cache, since this
is considered
> > > "unsafe"
> > > > > >> > migration, more info here:
> > > > > >> > > > https://doc.opensuse.org/documentation/leap/
> virtualization/
> > > > > >> > > html/book.virt/
> > > > > >> > > > cha.cachemodes.html#sec.cache.mode.live.migration
> > > > > >> > > >
> > > > > >> > > > or in short:
> > > > > >> > > > "
> > > > > >> > > > The libvirt management layer includes checks
for migration
> > > > > >> > > > compatibility based on several factors. If
the guest
> storage
> > > is
> > > > > >> > > > hosted on a clustered file system, is read-only
or is
> marked
> > > > > >> > > > shareable, then the cache mode is ignored
when determining
> > if
> > > > > >> > > > migration can be allowed. Otherwise libvirt
will not allow
> > > > > migration
> > > > > >> > > > unless the cache mode is set to none. However,
this
> > > restriction
> > > > > can
> > > > > >> > > > be overridden with the “unsafe” option
to the migration
> > APIs,
> > > > > which
> > > > > >> > > > is also supported by virsh, as for example
in
> > > > > >> > > >
> > > > > >> > > > virsh migrate --live --unsafe
> > > > > >> > > > "
> > > > > >> > > >
> > > > > >> > > > Cheers
> > > > > >> > > > Andrija
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On 20 February 2018 at 11:24, Paul Angus
<
> > > > > paul.angus@shapeblue.com>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > >> Hi Wido,
> > > > > >> > > >>
> > > > > >> > > >> This is for KVM (with Ceph backend as
it happens), the
> API
> > > > > >> > > >> documentation is out of sync with UI
capabilities, so I'm
> > > > trying
> > > > > to
> > > > > >> > > >> figure out if we
> > > > > >> > > >> *should* be able to set cacheMode for
root disks.  It
> seems
> > > to
> > > > > make
> > > > > >> > > quite a
> > > > > >> > > >> difference to performance.
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >> paul.angus@shapeblue.com
> > > > > >> > > >> www.shapeblue.com
> > > > > >> > > >> 53 Chandos Place, Covent Garden, London
 WC2N 4HSUK
> > > @shapeblue
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >> -----Original Message-----
> > > > > >> > > >> From: Wido den Hollander [mailto:wido@widodh.nl]
> > > > > >> > > >> Sent: 20 February 2018 09:03
> > > > > >> > > >> To: dev@cloudstack.apache.org
> > > > > >> > > >> Subject: Re: Caching modes
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >> On 02/20/2018 09:46 AM, Paul Angus wrote:
> > > > > >> > > >> > Hey guys,
> > > > > >> > > >> >
> > > > > >> > > >> > Can anyone shed any light on write
caching in
> CloudStack?
> > > > > >> >  cacheMode
> > > > > >> > > >> is available through the UI for data
disks (but not root
> > > > disks),
> > > > > >> but
> > > > > >> > not
> > > > > >> > > >> documented as an API option for data
or root disks
> > (although
> > > is
> > > > > >> > > documented
> > > > > >> > > >> as a response for data disks).
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >> What hypervisor?
> > > > > >> > > >>
> > > > > >> > > >> In case of KVM it's passed down to XML
which then passes
> it
> > > to
> > > > > >> > Qemu/KVM
> > > > > >> > > >> which then handles the caching.
> > > > > >> > > >>
> > > > > >> > > >> The implementation varies per hypervisor,
so that should
> be
> > > the
> > > > > >> > > question.
> > > > > >> > > >>
> > > > > >> > > >> Wido
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >> > #huh?
> > > > > >> > > >> >
> > > > > >> > > >> > thanks
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> > paul.angus@shapeblue.com
> > > > > >> > > >> > www.shapeblue.com
> > > > > >> > > >> > 53 Chandos Place, Covent Garden,
London  WC2N 4HSUK
> > > > @shapeblue
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > >
> > > > > >> > > > Andrija Panić
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > >
> > > > > >> > > Andrija Panić
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Rafael Weingärtner
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Rafael Weingärtner
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Andrija Panić
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Andrija Panić
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>
>
>
> --
>
> Andrija Panić
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message