incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: Vmware CPU Cap
Date Wed, 23 May 2012 20:23:44 GMT
Thanks for your input, I find another bug in cloudstack...
Let's dive little bit deeper into how the CPU allocation works.
VMware(the same for other hypervisors(Xen/KVM)) uses proportional share based scheduling algorithm([1],[2]).
It means
"If a virtual machine has twice as many shares of a resource as another virtual machine, it
is entitled to consume twice as much of that resource when these two virtual machines are
competing for resources."
There are two questions:
1. Where is the share coming from? In Cloudstack, the share is calculated from (CPU MHz *
CPU cores). 
2. How the share of a VM is mapped to the physical CPU by the hypervisor? Of cause, it depends
on the hypervisor implementation, but there are some general rules that we need to follow,
as we are living in the real world:)
   Rule 1: num of cpu core of a VM should be <= num of cpu core of the hypervisor host.
It doesn't make sense to running a 12 core VM on a single core host, even some hypervisors,
such as KVM, can do that. Hypervisor will be busy at scheduling VCPU(the VCPU context switch
is much heavier than process context switch), thus bad performance for the VM. 
   Rule 2: The frequency of a VCPU should be <= the frequency of the host hypervisor host.
Can you run an one core * 12Ghz VM on a 3Ghz * 4 core physical CPU? Nope, in an unit time,
the max freq of VCPU can get is the max freq of physical cpu core, that's the physical LAW.
   Rule 3: the total share of VMs <= total share of host hypervisor(in case of no CPU overcommit).
    

Currently, in cloudstack host allocator, only rule 3 is taken into consideration.

[1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM-resource.pdf
[2] http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf

> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Tuesday, May 22, 2012 5:41 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
> 
> Hi,
> 
> I still don't think this is an issue as the CPU Mhz limit and the
> number of cores are independent.
> CPU manufacturers sell 2,4,6 cores at 3Ghz and not 6,12,18Ghz CPUs.
> 
> So I think it is good how it works but the "number_of_cores*Mhz" while
> allocating should not multiply so that is the bug :)
> What vmware does with multiplying the cores with the core speed is bad
> as I can't have a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know
> where I'm coming from....
> 
> Regards
> 
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: 23 May 2012 01:27
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: Vmware CPU Cap
> 
> Hi,
> 
> I have confused myself too because if I have a look the database and
> dump the service_offerings table the limit for me is set to
> 2000,3000,4000 and not 1000.
> So a Mhz limit set to 4000 and 4 cores will end up as a quad core box
> at 4Ghz. I remember now I had to set CPU overprovisioning to 4 as
> CloudStack took away 4x4Ghz=16Ghz of the available CPU.
> So Cloudstack sets the VM CPU Mhz limit what the actual limit is set to
> in the offering but does not multiply the Mhz limit by the number of
> cores when setting the limit on vmware. However takes away
> "number_of_cores*Mhz limit" from the available CPU capacity when
> allocating.
> 
> I'm getting confused myself so I'm not sure if this is bug now or not
> either :) I'm using 3.0.3
> 
> Regards
> 
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: 23 May 2012 00:32
> To: cloudstack-dev@incubator.apache.org; Tamas Monos
> Subject: RE: Vmware CPU Cap
> 
> But in Diego'case, the limit is set as 1000Mhz, while his service
> offering is 1000Mhz * 2 cores.
> Which version of cloudstack are you using? Maybe it's a regression.
> 
> > -----Original Message-----
> > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > Sent: Tuesday, May 22, 2012 4:26 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Hi,
> >
> > I have a service offering with 1000Mhz limit and 4 cpu cores. That
> > sums up to 4000Mhz.
> > Cloudstack sets the limit to 4Ghz on the virtual machine and when I
> > put load on it vmware balances the load between the 4 cores allowing
> > them to use 1000Mhz each.
> > I do not see any bugs here.
> >
> > Apologies if the "meant per CPU core" was incorrect. What I meant is
> > described above.
> >
> > Regards
> >
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: 23 May 2012 00:05
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: Vmware CPU Cap
> >
> > Nope, from the document, the limit is set on whole vm:
> > CPU Limits
> >
> > When a CPU Limit is set on a virtual machine resource settings, the
> > virtual machine is deliberately held from being scheduled to a PCPU
> > when it has used up its allocated CPU resource. This happens
> > regardless of the CPU utilization. If the limit is set to 500MHz, the
> > virtual machine is descheduled from the PCPU and has to wait before
> it
> > is allowed to be scheduled again. As such, the virtual machine might
> > experience performance degradation.
> >
> > Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed
> > the specified limit. For example, 4 vCPU virtual machine with a limit
> > of 1200MHz and equal load among vCPUs would result in a max of 300MHz
> > per vCPU.
> >
> >
> http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup
> > =
> > true&languageId=&externalID=1033115
> >
> > > -----Original Message-----
> > > From: Tamas Monos [mailto:tamasm@veber.co.uk]
> > > Sent: Tuesday, May 22, 2012 3:43 PM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: RE: Vmware CPU Cap
> > >
> > > Hi,
> > >
> > > No I don't think this is a bug. When you set 1000Mhz as CPU cap
> that
> > > is meant per core. vmWare will limit each CPU core to 1000Mhz.
> > > As you gave 2 CPU cores that is 2000Mhz effective. That is how
> > > vmware works.
> > >
> > > I have setup my offerings all to 1000Mhz as speed and just
> > > increasing the number of cores.
> > > 1 core ends up being 1x1000Mhz
> > > 2 core ends up being 2x1000Mhz=2000Mhz ...
> > > ...
> > >
> > > I'm actually using it in production and works quite well as
> > Cloudstack
> > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and vmware
> > cleverly
> > > balances between the cores as you put load on it and not letting
> any
> > > of the cores above the set limit of 1000Mhz.
> > >
> > >
> > > Regards
> > >
> > > -----Original Message-----
> > > From: Diego Spinola Castro [mailto:spinolacastro@gmail.com]
> > > Sent: 22 May 2012 19:34
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: Re: Vmware CPU Cap
> > >
> > > I forgot the link:
> > > http://imageshack.us/photo/my-images/848/cpulimit.png/
> > >
> > > 2012/5/22 Diego Spinola Castro <spinolacastro@gmail.com>
> > >
> > > > I believe that is a bug with cpu cap and vmware.
> > > >
> > > > To reproduce:
> > > >
> > > > Create a offering with 2 cores and 1000mhz.
> > > > Enable CPU CAP.
> > > >
> > > > After created instance , cs create a vm with 2 cores and 1000mhz
> > > > of
> > > limit.
> > > >
> > > > I don't know for sure if is a bug, but vmware gives 1000mhz
> shared
> > > > with cores.
> > > >
> > > >
> > > >
> > > > Diego
> > > >
> >
> >
> 
> 
> 
> 


Mime
View raw message