cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: Vmware CPU Cap
Date Fri, 25 May 2012 17:36:24 GMT
Two bugs are been created:
http://bugs.cloudstack.org/browse/CS-15095
http://bugs.cloudstack.org/browse/CS-15094
Let me fix it on the master branch, but not sure will it be pushed into next release(3.0.3).

> -----Original Message-----
> From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> Sent: Friday, May 25, 2012 5:07 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
>
> Edison. What we are going to do? I believe this is a important path
> inspite
> of cs isnt using the right way of resource allocation.
>
> []'s
> Diego Castro
> Em 23/05/2012 19:29, "Edison Su" <Edison.su@citrix.com> escreveu:
>
> > ...Sorry, I am not read your last email carefully...
> > As I said before, in VMware, the limit parameter is set on the whole
> VM,
> > not per core. So we need to set the limit as " core_number*core_speed
> ",
> > which means
> > the VM as a whole will not use as much as " core_number*core_speed "
> > shares. It doesn't break the rule 2.
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Wednesday, May 23, 2012 3:20 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > I'm afraid I don't follow.
> > > I was talking about multiple core VMs and not single core ones.
> > > I said if I have a 4 core VM with 3Ghz core speed and I run 4
> cpuburn
> > > threads I could get approximately the performance of a single core
> at
> > > 12Ghz.
> > >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: 23 May 2012 23:06
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Single core means there is only one scheduling entity, that
> hypervisor
> > > can schedule for the VM, so there is no parallel processing at all.
> > > Hypervisor can not schedule a single core VM to multiple physical
> cpu
> > > cores at the same time.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Wednesday, May 23, 2012 2:59 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > Before we would touch that rule 2:
> > > > While utilizing 12Ghz on a single core is not possible it is
> possible
> > > > to have effective 12Ghz on 4 cores at 3Ghz via parallel
> processing.
> > > > If I start 4 cpuburn threads on a quad core it will use all four
> 3Ghz
> > > > cores flat out "giving similar speed as 1 core@12Ghz would be"*.
> As
> > > > vmware uses the limit parameter as limit to the whole CPU
> utilization
> > > > rule 2 approach would prevent me effectively utilizing the CPU on
> > > > vmware because if we say the CPU speed in Mhz cannot be larger
> than
> > > > the core speed I would be limited to an effective speed of core-
> > > > speed/number_of_cores. This would mean  3000/4=750Mhz per core on
> a
> > > > 4vCPU VM. If you set "core_number*core_speed" as limit you
> already
> > > > broke rule 2 as the CPU is set higher than its core speed.
> > > >
> > > > I believe it good as it is now.
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Prachi Damle [mailto:Prachi.Damle@citrix.com]
> > > > Sent: 23 May 2012 22:36
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Yeah, rule#1 and #3 are considered by the current Host Allocator;
> > > > rule#2 to compare just the speed is missing.
> > > >
> > > > Prachi
> > > >
> > > > -----Original Message-----
> > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > Sent: Wednesday, May 23, 2012 2:06 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Checked the code, I think we miss rule 2 in allocator.
> > > >
> > > > Anthony
> > > >
> > > > > -----Original Message-----
> > > > > From: Anthony Xu [mailto:Xuefei.Xu@citrix.com]
> > > > > Sent: Wednesday, May 23, 2012 1:50 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > I'm pretty sure we have that before we redesign allocator.
> > > > >
> > > > > Anthony
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > Sent: Wednesday, May 23, 2012 1:44 PM
> > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > Subject: RE: Vmware CPU Cap
> > > > > >
> > > > > > I'm surprised.  I definitely recall having that conversation
> > > > > > before about never exceeding the physical limits of the
> > > > > > hypervisor.  I'm pretty sure we have test cases against that
> as
> > > well.
> > > > > >
> > > > > > --Alex
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > Sent: Wednesday, May 23, 2012 1:39 PM
> > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > >
> > > > > > > I think it's a general issue for all the hypervisors we
> > > supported.
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > > > > > > Sent: Wednesday, May 23, 2012 1:32 PM
> > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > >
> > > > > > > > That's because no one wrote a host allocator for VmWare.
> > > > > > > >
> > > > > > > > The relationship between DeploymentPlanner and
> HostAllocator
> > > > and
> > > > > > > > StoragePoolAllocator is as follows:
> > > > > > > >
> > > > > > > > DeploymentPlanner deals with heuristics and affinity
> rules of
> > > > > > placing
> > > > > > > > a VM.  Once it determines a set of hosts that matches,
it
> > > then
> > > > > asks
> > > > > > > > the HostAllocator to work on the limitations of the
type
> of
> > > > > > hypervisor.
> > > > > > > > There was never a HostAllocator written for VmWare.
 It
> just
> > > > > > > > uses
> > > > > > the
> > > > > > > > one that was written originally for XenServer.  Hence
the
> bug.
> > > > > > > >
> > > > > > > > There's similar differentiation for DeploymentPlanner
and
> > > > > > > > StoragePoolAllocator and similar bugs exists, especially
> if
> > > > > someone
> > > > > > > > adds a completely new type of storage pool but decides
to
> > > just
> > > > > > reuse
> > > > > > > > the current set of StoragePoolAllocators.
> > > > > > > >
> > > > > > > > --Alex
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM
> > > > > > > > > To: cloudstack-dev@incubator.apache.org;
> > > 'tamasm@veber.co.uk'
> > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > >
> > > > > > > > > Thanks for your input, I find another bug in
> cloudstack...
> > > > > > > > > Let's dive little bit deeper into how the CPU
> allocation
> > > > works.
> > > > > > > > > VMware(the same for other hypervisors(Xen/KVM))
uses
> > > > > proportional
> > > > > > > > share
> > > > > > > > > based scheduling algorithm([1],[2]). It means
"If a
> virtual
> > > > > > machine
> > > > > > > > has twice
> > > > > > > > > as many shares of a resource as another virtual
machine,
> it
> > > > is
> > > > > > > > entitled to
> > > > > > > > > consume twice as much of that resource when these
two
> > > > > > > > > virtual
> > > > > > > > machines
> > > > > > > > > are competing for resources."
> > > > > > > > > There are two questions:
> > > > > > > > > 1. Where is the share coming from? In Cloudstack,
the
> share
> > > > is
> > > > > > > > calculated
> > > > > > > > > from (CPU MHz * CPU cores).
> > > > > > > > > 2. How the share of a VM is mapped to the physical
CPU
> by
> > > > > > > > > the
> > > > > > > > hypervisor?
> > > > > > > > > Of cause, it depends on the hypervisor implementation,
> but
> > > > > there
> > > > > > are
> > > > > > > > some
> > > > > > > > > general rules that we need to follow, as we are
living
> in
> > > > > > > > > the
> > > > > > real
> > > > > > > > world:)
> > > > > > > > >    Rule 1: num of cpu core of a VM should be
<= num of
> cpu
> > > > > > > > > core
> > > > > > of
> > > > > > > > the
> > > > > > > > > hypervisor host. It doesn't make sense to running
a 12
> core
> > > > VM
> > > > > on
> > > > > > a
> > > > > > > > single
> > > > > > > > > core host, even some hypervisors, such as KVM,
can do
> that.
> > > > > > > > Hypervisor will
> > > > > > > > > be busy at scheduling VCPU(the VCPU context switch
is
> much
> > > > > > heavier
> > > > > > > > than
> > > > > > > > > process context switch), thus bad performance
for the
> VM.
> > > > > > > > >    Rule 2: The frequency of a VCPU should be
<= the
> > > > > > > > > frequency
> > > > > of
> > > > > > the
> > > > > > > > host
> > > > > > > > > hypervisor host. Can you run an one core * 12Ghz
VM on
> a
> > > > > > > > > 3Ghz
> > > > > > > > > *
> > > > > 4
> > > > > > > > core
> > > > > > > > > physical CPU? Nope, in an unit time, the max
freq of
> VCPU
> > > > > > > > > can
> > > > > get
> > > > > > is
> > > > > > > > the max
> > > > > > > > > freq of physical cpu core, that's the physical
LAW.
> > > > > > > > >    Rule 3: the total share of VMs <= total
share of
> host
> > > > > > > > hypervisor(in case of
> > > > > > > > > no CPU overcommit).
> > > > > > > > >
> > > > > > > > > Currently, in cloudstack host allocator, only
rule 3 is
> > > > > > > > > taken
> > > > > > into
> > > > > > > > consideration.
> > > > > > > > >
> > > > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-
> > > > > resource.pdf
> > > > > > > > > [2]
> > > > > > > > >
> > > > > > >
> http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I still don't think this is an issue as
the CPU Mhz
> limit
> > > > > > > > > > and
> > > > > > the
> > > > > > > > > > number of cores are independent.
> > > > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz
and not
> > > > > > > > > > 6,12,18Ghz
> > > > > > CPUs.
> > > > > > > > > >
> > > > > > > > > > So I think it is good how it works but the
> > > > > > "number_of_cores*Mhz"
> > > > > > > > while
> > > > > > > > > > allocating should not multiply so that is
the bug :)
> What
> > > > > > vmware
> > > > > > > > does
> > > > > > > > > > with multiplying the cores with the core
speed is bad
> as
> > > I
> > > > > > can't
> > > > > > > > have
> > > > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core
if you know
> where
> > > > > > > > > > I'm
> > > > > > > > coming
> > > > > > > > > > from....
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > Sent: 23 May 2012 01:27
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have confused myself too because if I
have a look
> the
> > > > > > database
> > > > > > > > and
> > > > > > > > > > dump the service_offerings table the limit
for me is
> set
> > > > > > > > > > to
> > > > > > > > > > 2000,3000,4000 and not 1000.
> > > > > > > > > > So a Mhz limit set to 4000 and 4 cores will
end up as
> a
> > > > quad
> > > > > > core
> > > > > > > > box
> > > > > > > > > > at 4Ghz. I remember now I had to set CPU
> overprovisioning
> > > > to
> > > > > 4
> > > > > > as
> > > > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the
available
> CPU.
> > > > > > > > > > So Cloudstack sets the VM CPU Mhz limit
what the
> actual
> > > > > > > > > > limit
> > > > > > is
> > > > > > > > set
> > > > > > > > > > to in the offering but does not multiply
the Mhz
> limit by
> > > > > > > > > > the
> > > > > > > > number
> > > > > > > > > > of cores when setting the limit on vmware.
However
> takes
> > > > > > > > > > away "number_of_cores*Mhz limit" from the
available
> CPU
> > > > > > > > > > capacity
> > > > > > when
> > > > > > > > > > allocating.
> > > > > > > > > >
> > > > > > > > > > I'm getting confused myself so I'm not sure
if this
> is
> > > bug
> > > > > now
> > > > > > or
> > > > > > > > not
> > > > > > > > > > either :) I'm using 3.0.3
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > Sent: 23 May 2012 00:32
> > > > > > > > > > To: cloudstack-dev@incubator.apache.org;
Tamas Monos
> > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > >
> > > > > > > > > > But in Diego'case, the limit is set as 1000Mhz,
while
> his
> > > > > > service
> > > > > > > > > > offering is 1000Mhz * 2 cores.
> > > > > > > > > > Which version of cloudstack are you using?
Maybe it's
> a
> > > > > > regression.
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have a service offering with 1000Mhz
limit and 4
> cpu
> > > > > cores.
> > > > > > > > That
> > > > > > > > > > > sums up to 4000Mhz.
> > > > > > > > > > > Cloudstack sets the limit to 4Ghz on
the virtual
> > > machine
> > > > > and
> > > > > > > > > > > when
> > > > > > > > I
> > > > > > > > > > > put load on it vmware balances the
load between the
> 4
> > > > > > > > > > > cores
> > > > > > > > allowing
> > > > > > > > > > > them to use 1000Mhz each.
> > > > > > > > > > > I do not see any bugs here.
> > > > > > > > > > >
> > > > > > > > > > > Apologies if the "meant per CPU core"
was incorrect.
> > > > > > > > > > > What I meant
> > > > > > > > is
> > > > > > > > > > > described above.
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > > > > > > > > Sent: 23 May 2012 00:05
> > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > >
> > > > > > > > > > > Nope, from the document, the limit
is set on whole
> vm:
> > > > > > > > > > > CPU Limits
> > > > > > > > > > >
> > > > > > > > > > > When a CPU Limit is set on a virtual
machine
> resource
> > > > > > settings,
> > > > > > > > the
> > > > > > > > > > > virtual machine is deliberately held
from being
> > > > > > > > > > > scheduled
> > > > > to
> > > > > > a
> > > > > > > > PCPU
> > > > > > > > > > > when it has used up its allocated CPU
resource.
> This
> > > > > happens
> > > > > > > > > > > regardless of the CPU utilization.
If the limit is
> set
> > > > > > > > > > > to 500MHz, the virtual machine is descheduled
from
> the
> > > > > > > > > > > PCPU
> > > > > and
> > > > > > has
> > > > > > > > > > > to wait before
> > > > > > > > > > it
> > > > > > > > > > > is allowed to be scheduled again. As
such, the
> virtual
> > > > > > machine
> > > > > > > > might
> > > > > > > > > > > experience performance degradation.
> > > > > > > > > > >
> > > > > > > > > > > Note: For an SMP virtual machine, the
sum of all
> vCPUs
> > > > > cannot
> > > > > > > > exceed
> > > > > > > > > > > the specified limit. For example, 4
vCPU virtual
> > > machine
> > > > > with
> > > > > > a
> > > > > > > > > > > limit of 1200MHz and equal load among
vCPUs would
> > > result
> > > > > > > > > > > in
> > > > > a
> > > > > > > > > > > max
> > > > > > > > of
> > > > > > > > > > > 300MHz per vCPU.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop
> > > > > > > > > u
> > > > > > > > > p
> > > > > > > > > > > =
> > > > > > > > > > > true&languageId=&externalID=1033115
> > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43
PM
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: RE: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > No I don't think this is a bug.
When you set
> 1000Mhz
> > > > > > > > > > > > as
> > > > > CPU
> > > > > > > > > > > > cap
> > > > > > > > > > that
> > > > > > > > > > > > is meant per core. vmWare will
limit each CPU
> core to
> > > > > > 1000Mhz.
> > > > > > > > > > > > As you gave 2 CPU cores that is
2000Mhz effective.
> > > > > > > > > > > > That
> > > > > is
> > > > > > how
> > > > > > > > > > > > vmware works.
> > > > > > > > > > > >
> > > > > > > > > > > > I have setup my offerings all
to 1000Mhz as speed
> and
> > > > > just
> > > > > > > > > > > > increasing the number of cores.
> > > > > > > > > > > > 1 core ends up being 1x1000Mhz
> > > > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz
...
> > > > > > > > > > > > ...
> > > > > > > > > > > >
> > > > > > > > > > > > I'm actually using it in production
and works
> quite
> > > > well
> > > > > as
> > > > > > > > > > > Cloudstack
> > > > > > > > > > > > "allocates" 4000Mhz when I'm using
4x1000Mhz
> cores
> > > and
> > > > > > vmware
> > > > > > > > > > > cleverly
> > > > > > > > > > > > balances between the cores as
you put load on it
> and
> > > > not
> > > > > > > > letting
> > > > > > > > > > any
> > > > > > > > > > > > of the cores above the set limit
of 1000Mhz.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regards
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Diego Spinola Castro
> > > > > [mailto:spinolacastro@gmail.com]
> > > > > > > > > > > > Sent: 22 May 2012 19:34
> > > > > > > > > > > > To: cloudstack-dev@incubator.apache.org
> > > > > > > > > > > > Subject: Re: Vmware CPU Cap
> > > > > > > > > > > >
> > > > > > > > > > > > I forgot the link:
> > > > > > > > > > > > http://imageshack.us/photo/my-
> > > images/848/cpulimit.png/
> > > > > > > > > > > >
> > > > > > > > > > > > 2012/5/22 Diego Spinola Castro
> > > > <spinolacastro@gmail.com>
> > > > > > > > > > > >
> > > > > > > > > > > > > I believe that is a bug with
cpu cap and vmware.
> > > > > > > > > > > > >
> > > > > > > > > > > > > To reproduce:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Create a offering with 2
cores and 1000mhz.
> > > > > > > > > > > > > Enable CPU CAP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > After created instance ,
cs create a vm with 2
> > > cores
> > > > > and
> > > > > > > > 1000mhz
> > > > > > > > > > > > > of
> > > > > > > > > > > > limit.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't know for sure if
is a bug, but vmware
> gives
> > > > > > 1000mhz
> > > > > > > > > > shared
> > > > > > > > > > > > > with cores.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Diego
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > >
> > > >
> > >
> > >
> >
> >

Mime
View raw message