incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: Vmware CPU Cap
Date Wed, 23 May 2012 20:39:16 GMT
I think it's a general issue for all the hypervisors we supported. 

> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Wednesday, May 23, 2012 1:32 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
> 
> That's because no one wrote a host allocator for VmWare.
> 
> The relationship between DeploymentPlanner and HostAllocator and
> StoragePoolAllocator is as follows:
> 
> DeploymentPlanner deals with heuristics and affinity rules of placing a
> VM.  Once it determines a set of hosts that matches, it then asks the
> HostAllocator to work on the limitations of the type of hypervisor.
> There was never a HostAllocator written for VmWare.  It just uses the
> one that was written originally for XenServer.  Hence the bug.
> 
> There's similar differentiation for DeploymentPlanner and
> StoragePoolAllocator and similar bugs exists, especially if someone
> adds a completely new type of storage pool but decides to just reuse
> the current set of StoragePoolAllocators.
> 
> --Alex
> 
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Wednesday, May 23, 2012 1:24 PM
> > To: cloudstack-dev@incubator.apache.org; 'tamasm@veber.co.uk'
> > Subject: RE: Vmware CPU Cap
> >
> > Thanks for your input, I find another bug in cloudstack...
> > Let's dive little bit deeper into how the CPU allocation works.
> > VMware(the same for other hypervisors(Xen/KVM)) uses proportional
> share
> > based scheduling algorithm([1],[2]). It means "If a virtual machine
> has twice
> > as many shares of a resource as another virtual machine, it is
> entitled to
> > consume twice as much of that resource when these two virtual
> machines
> > are competing for resources."
> > There are two questions:
> > 1. Where is the share coming from? In Cloudstack, the share is
> calculated
> > from (CPU MHz * CPU cores).
> > 2. How the share of a VM is mapped to the physical CPU by the
> hypervisor?
> > Of cause, it depends on the hypervisor implementation, but there are
> some
> > general rules that we need to follow, as we are living in the real
> world:)
> >    Rule 1: num of cpu core of a VM should be <= num of cpu core of
> the
> > hypervisor host. It doesn't make sense to running a 12 core VM on a
> single
> > core host, even some hypervisors, such as KVM, can do that.
> Hypervisor will
> > be busy at scheduling VCPU(the VCPU context switch is much heavier
> than
> > process context switch), thus bad performance for the VM.
> >    Rule 2: The frequency of a VCPU should be <= the frequency of the
> host
> > hypervisor host. Can you run an one core * 12Ghz VM on a 3Ghz * 4
> core
> > physical CPU? Nope, in an unit time, the max freq of VCPU can get is
> the max
> > freq of physical cpu core, that's the physical LAW.
> >    Rule 3: the total share of VMs <= total share of host
> hypervisor(in case of
> > no CPU overcommit).
> >
> > Currently, in cloudstack host allocator, only rule 3 is taken into
> consideration.
> >
> > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
> > [2]
> > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Tuesday, May 22, 2012 5:41 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > I still don't think this is an issue as the CPU Mhz limit and the
> > > number of cores are independent.
> > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
> > >
> > > So I think it is good how it works but the "number_of_cores*Mhz"
> while
> > > allocating should not multiply so that is the bug :) What vmware
> does
> > > with multiplying the cores with the core speed is bad as I can't
> have
> > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where I'm
> coming
> > > from....
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: 23 May 2012 01:27
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > I have confused myself too because if I have a look the database
> and
> > > dump the service_offerings table the limit for me is set to
> > > 2000,3000,4000 and not 1000.
> > > So a Mhz limit set to 4000 and 4 cores will end up as a quad core
> box
> > > at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> > > CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> > > So Cloudstack sets the VM CPU Mhz limit what the actual limit is
> set
> > > to in the offering but does not multiply the Mhz limit by the
> number
> > > of cores when setting the limit on vmware. However takes away
> > > "number_of_cores*Mhz limit" from the available CPU capacity when
> > > allocating.
> > >
> > > I'm getting confused myself so I'm not sure if this is bug now or
> not
> > > either :) I'm using 3.0.3
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > Sent: 23 May 2012 00:32
> > > To: cloudstack-dev@incubator.apache.org; Tamas Monos
> > > Subject: RE: Vmware CPU Cap
> > >
> > > But in Diego'case, the limit is set as 1000Mhz, while his service
> > > offering is 1000Mhz * 2 cores.
> > > Which version of cloudstack are you using? Maybe it's a regression.
> > >
> > > > -----Original Message-----
> > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > Sent: Tuesday, May 22, 2012 4:26 PM
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Hi,
> > > >
> > > > I have a service offering with 1000Mhz limit and 4 cpu cores.
> That
> > > > sums up to 4000Mhz.
> > > > Cloudstack sets the limit to 4Ghz on the virtual machine and when
> I
> > > > put load on it vmware balances the load between the 4 cores
> allowing
> > > > them to use 1000Mhz each.
> > > > I do not see any bugs here.
> > > >
> > > > Apologies if the "meant per CPU core" was incorrect. What I meant
> is
> > > > described above.
> > > >
> > > > Regards
> > > >
> > > > -----Original Message-----
> > > > From: Edison Su [mailto:Edison.su@citrix.com]
> > > > Sent: 23 May 2012 00:05
> > > > To: cloudstack-dev@incubator.apache.org
> > > > Subject: RE: Vmware CPU Cap
> > > >
> > > > Nope, from the document, the limit is set on whole vm:
> > > > CPU Limits
> > > >
> > > > When a CPU Limit is set on a virtual machine resource settings,
> the
> > > > virtual machine is deliberately held from being scheduled to a
> PCPU
> > > > when it has used up its allocated CPU resource. This happens
> > > > regardless of the CPU utilization. If the limit is set to 500MHz,
> > > > the virtual machine is descheduled from the PCPU and has to wait
> > > > before
> > > it
> > > > is allowed to be scheduled again. As such, the virtual machine
> might
> > > > experience performance degradation.
> > > >
> > > > Note: For an SMP virtual machine, the sum of all vCPUs cannot
> exceed
> > > > the specified limit. For example, 4 vCPU virtual machine with a
> > > > limit of 1200MHz and equal load among vCPUs would result in a max
> of
> > > > 300MHz per vCPU.
> > > >
> > > >
> > >
> > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popu
> > p
> > > > =
> > > > true&languageId=&externalID=1033115
> > > >
> > > > > -----Original Message-----
> > > > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: RE: Vmware CPU Cap
> > > > >
> > > > > Hi,
> > > > >
> > > > > No I don't think this is a bug. When you set 1000Mhz as CPU cap
> > > that
> > > > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > > > vmware works.
> > > > >
> > > > > I have setup my offerings all to 1000Mhz as speed and just
> > > > > increasing the number of cores.
> > > > > 1 core ends up being 1x1000Mhz
> > > > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > > > ...
> > > > >
> > > > > I'm actually using it in production and works quite well as
> > > > Cloudstack
> > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > > > cleverly
> > > > > balances between the cores as you put load on it and not
> letting
> > > any
> > > > > of the cores above the set limit of 1000Mhz.
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > -----Original Message-----
> > > > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > > > Sent: 22 May 2012 19:34
> > > > > To: cloudstack-dev@incubator.apache.org
> > > > > Subject: Re: Vmware CPU Cap
> > > > >
> > > > > I forgot the link:
> > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > > > >
> > > > > 2012/5/22 Diego Spinola Castro <spinolacastro@gmail.com>
> > > > >
> > > > > > I believe that is a bug with cpu cap and vmware.
> > > > > >
> > > > > > To reproduce:
> > > > > >
> > > > > > Create a offering with 2 cores and 1000mhz.
> > > > > > Enable CPU CAP.
> > > > > >
> > > > > > After created instance , cs create a vm with 2 cores and
> 1000mhz
> > > > > > of
> > > > > limit.
> > > > > >
> > > > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> > > shared
> > > > > > with cores.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Diego
> > > > > >
> > > >
> > > >
> > >
> > >
> > >
> > >


Mime
View raw message